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FOREWORD 


The  Joint  Test  and  Evaluation  (JT&E)  Program  has  been  established  to  evaluate  concepts  and 
address  needs  and  issues  that  occur  in  joint  military  environments.  The  program  includes  the 
nomination  process;  the  Joint  Feasibility  Study  (JFS),  which  determines  whether  selected 
nominations  are  needed  and  feasible;  and,  when  such  a  determination  is  made,  the  execution  of  a 
JT&E  by  an  Office  of  the  Secretary  of  Defense  (OSD)  chartered  Joint  Test  Force  (JTF). 

DoD  Directive  5000.3-M-4  describes  the  JT&E  Program  process,  identifies  the  principle 
participants  and  their  responsibilities,  and  outlines  the  framework  within  which  each  Service 
supports  the  program.  Service  support  is  described  in  detail  in  applicable  Service  directives  and 
the  Memorandum  of  Agreement  on  Multi-Service  Operational  Test  and  Evaluation  and  Joint  Test 
and  Evaluation  which  is  maintained  by  the  Services'  Operational  Test  Agencies  (OTA). 

This  handbook  is  one  of  three  that  discuss  testing  and  evaluation  in  the  joint  military 
environment:  the  Joint  Test  and  Evaluation  Nomination  Handbook;  the  Joint  Feasibility  Study 
Handbook  and  the  Joint  Test  and  Evaluation  Handbook.  These  companion  volumes  are  designed 
to  provide  OSD  guidance  and  direction,  information,  references,  and  procedures  to  joint  test 
directors  and  their  staffs.  These  handbooks  provide  "how  to"  lessons  learned  based  primarily 
upon  inputs  from  personnel  who  have  participated  in  JFSs  and  JTFs  in  various  capacities. 

The  proponent  of  this  handbook  is  the  Deputy  Director,  Test  and  Evaluation. 

Recommended  changes  or  suggestions  for  additions  should  be  forwarded  to: 

Deputy  Director,  Test  and  Evaluation 

3110-Defense  Pentagon,  Room  3D1080 

Washington,  D.C.  20301-3110 


uoA-W&\ 

Richard  R.  Ledesma  ^  ^ 

Deputy  Director,  Test  and  Evaluation 
OUSD  (A&T) 
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CHAPTER  1 

JOINT  TEST  AND  EVALUATION  PROGRAM  BACKGROUND 


A.  INTRODUCTION 

The  Department  of  Defense  (DoD)  Joint 
Test  and  Evaluation  (JT&E)  program  originated 
in  response  to  the  1970  Blue  Ribbon  Defense 
Panel  Report.  The  panel  report  concluded  that 
"There  has  been,  and  is  currently,  no 
effective  means  for  conducting 
productive  joint  operations  tests  and 
evaluations.  The  fact  that  some  such 
efforts  heretofore  have  encountered 
difficulties  and  achieved  few  useful 
results  does  not  obviate  the  requirements 
for  much  needed  joint  operational  test 
and  evaluation."  The  report  recommended 
that  continuing  responsibility  for  joint  testing  be 
vested  in  an  Office  of  the  Secretary  of  Defense 
(OSD)  staff  element  dedicated  to  test  and 
evaluation  (T&E).  While  the  original  emphasis 
was  on  joint  operational  T&E,  evolving  multi- 
Service  needs  and  issues  have  led  to  an 
expansion  of  the  effort  to  include  joint 
developmental  as  well  as  operational  T&Es.  The 
JT&E  program  is  directed  by  the  Director,  Test, 
Systems  Engineering  and  Evaluation 
(D,T,SE&E). 

The  Deputy  Director,  Test  and  Evaluation 
(DDT&E)  manages  and  administers  the  JT&E 
program  and  develops  the  means  to  ensure  that 
productive  testing  is  accomplished.  The  policies 
and  details  in  the  JT&E  program  are  set  forth  in 
DoD  5000.3-M-4,  Joint  Test  and  Evaluation 
Procedures  Manual.  The  JT&E  Procedures 
Manual  provides  a  description  of  the  JT&E 
program,  OSD  and  Service  responsibilities 


relative  to  the  program,  and  defines  the  JT&E 
nomination  and  selection  process.  Key  to  the 
management  structure  is  the  independent  status  of 
the  OSD  JT&E  program.  That  is,  the  program  is 
structured  so  that  it  minimizes  the  influence  of 
Service  biases.  The  Feasibility  Study  Director 
(FSD)  and  Joint  Test  Director  (JTD)  report 
directly  to  OSD  for  direction  so  that  the  JT&E 
findings  are  approved  by  OSD.  Additionally, 
OSD  is  the  primary  source  of  JT&E  funding, 
thus  assuring  independence  of  program  control. 

While  Joint  Feasibility  Study  (JFS)  and 
Joint  Test  Force  (JTF)  activities  are  conducted  in 
accordance  with  established  joint  procedures, 
consistency  is  maintained  with  the  assigned 
mission  of  each  participating  Service. 
Responsibility  for  conducting  a  JFS  is  assigned 
to  a  lead  Service.  The  JFS  is  supported  by 
personnel  and  resources  from  the  designated 
participating  Services. 

B .  PURPOSE 

The  purpose  of  a  JT&E  is  to: 

•  Assess  the  interoperability  of  Service 
systems  in  joint  operations  and  explore 
potential  solutions  to  identified  problems. 

•  Evaluate  and  provide  recommendations  for 
improvements  in  joint  technical  and 
operational  concepts. 

•  Develop  and  validate  system  development 
and  testing  methodologies  having  multi- 
Service  application. 
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•  Evaluate  technical  or  operational 
performance  of  interrelated/interacting 
systems  under  realistic  joint  operational 
conditions. 

The  purpose  of  a  JFS  is  to  conduct  an  in- 
depth  analysis  of  a  JT&E  nomination  and  provide 
information  and  recommendations  upon  which 
the  Senior  Advisory  Council  (SAC)  and  the 
D,T,SE&E  will  base  decisions  relative  to  the 
chartering  of  a  JT&E.  This  information  will 
include  an  accurate  projection  of  the  resource 
requirements  (funding  from  OSD  and  the 
Services,  test  assets,  troops,  JT&E  personnel, 
contractor  support,  exercise  participants, 
administration,  comptroller,  accommodations, 
etc.)  that  must  be  provided  by  OSD  and  the 
Services.  The  JFS  will  focus  on  what  concepts 
or  issues  the  nominated  JT&E  will  resolve,  how 
the  JT&E  will  resolve  them,  whether  there  is  a 
broad  support  base  for  the  JT&E,  and  whether 
the  JT&E  is  needed  and  feasible.  The  lead  and 
participating  Service  points  of  contact  (POCs) 
will  use  the  JFS  results  to  pre-brief  the  SAC 
members  on  the  resource  requirements  if  a  JT&E 
is  being  recommended. 

C.  JT&E  PROGRAM 

The  JT&E  program  is  composed  of  three 
separate  but  closely  related  activities. 

1)  The  nomination,  coordination,  and 
approval  of  concepts  or  issues  to  be 
addressed  by  a  JT&E. 

2)  A  feasibility  study  to  determine  the  need 
and  feasibility  of  the  approved 
nominations. 


3)  The  execution  of  those  JT&Es  that  show 

potential  for  significant  improvements  in 

joint  capabilities. 

Figure  1-1  is  an  overview  of  the 
sequential  relationship  of  these  activities.  JT&E 
activities  are  sequentially  accomplished  with 
reviews  conducted  at  designated  milestones  to 
assure  that  OSD  and  Service  resources  are 
available  and  are  wisely  expended.  The  activities 
are  designed  to  be  expanded  in  both  scope  and 
level  of  detail  as  information  and  expertise  are 
developed,  thus  increasing  confidence  in 
program  decisions  at  the  designated  milestones. 
For  this  reason,  the  Feasibility  Study  Director 
(FSD)  and  staff  must  fully  understand  the  entire 
JT&E  program  in  order  to  be  able  to  refine  the 
concepts,  issues,  objectives,  plan  of  action, 
milestones,  and  estimate  of  resources  that  were 
developed  during  the  nomination  process.  The 
JFS  must  also  consider  JTF  requirements 
(location,  personnel,  and  resource  requirements) 
based  on  the  assumption  that  a  JTF  will  be 
chartered. 

The  JT&E  program  cycle  begins  each  year 
with  a  request  (call)  from  the  D,T,SE&E  to  the 
Services,  commanders  in  chief  (CINCs),  defense 
agencies,  the  joint  staff,  and  the  OSD  staff  for 
nominations.  This  call  is  usually  made  in 
October/November  with  the  stipulation  that 
proposed  nominations  be  forwarded  to  the 
D,T,SE&E  by  early  March.  The  CINCs, 
Services,  and  defense  agencies  develop  and 
refine  their  nominations  and  forward  them  in  the 
format  prescribed  in  DoD  5000.3-M-4. 
Figure  1-2  is  an  overview  of  those  activities 
that  are  involved  in  the  JT&E  nomination 
process.  Figure  1-3  relates  the  nomination 
process  and  major  JFS  activities  to  suggested 
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milestones.  Nominations  must  address  needs, 
concepts,  or  issues  that  involve  more  than  one 
Service  and  must  contain  an  explicit  purpose 
statement  with  expected  results  and  issues  to  be 
addressed.  Each  nomination  should  also  include: 

•  A  POC  and  how  to  contact  him  or  her. 

•  Anticipated  users  and  a  list  of  agencies/ 
organizations/CINCs  concurring  in  the 
nomination. 

•  A  projected  JT&E  schedule. 

•  The  identification  of  unique  requirements. 

•  Suggested  JT&E  participants. 

•  An  outline  of  the  test  concept. 


•  A  JFS  plan  of  action  and  milestones. 

•  An  estimate  of  required  funds. 

•  Known  or  projected  shortfalls. 

•  The  name  of  the  recommended  FSD. 

•  The  products  and  expected  benefits  in  terms 
of  an  increase  in  joint  warfighting 
capability. 

Upon  receipt  of  all  nominations,  a  JT&E 
Planning  Committee  (PC)  will  be  convened  to 
review  and  winnow  the  nominations  and  prepare 
the  highest  priority  candidates  for  SAC  review. 
The  JT&E  PC  is  mostly  composed  of  Service 
representatives  that  will  exchange  information  on 
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Figure  1-2.  JT&E  Nomination  and  JFS  Activities 


Service  positions,  coordinate  on  projected 
resource  requirements,  and  assist  in  the 
preparation  of  a  nomination  report  that  is  to  be 
presented  to  the  SAC.  The  makeup  and 
responsibilities  of  the  JT&E  PC  are  outlined  in 
DoD  5000.3-M-4.  The  JT&E  PC  joint  staff 
member  will  provide  joint  exercise  information 
during  the  nomination  review  process. 

DoD  5000.3-M-4  also  outlines  the 
membership,  duties,  and  responsibilities  of  the 
SAC.  The  SAC  reviews  the  JT&E  PC  reports 
and  develops  recommendations  relative  to  which 
candidates  should  be  approved,  prioritizes  the 
candidates,  and  recommends  which  candidates 
should  proceed  to  a  JFS  or  directly  to  a  JT&E 
conducted  by  a  JTF.  Based  on  the  SAC 
recommendations,  the  D,T,SE&E  approves 
candidates  for  execution,  designates  the  lead  and 


participating  Services,  considers  the  lead  Service 
recommendation  for  FSD,  appoints  the  FSD,  and 
directs  the  JFS  to  be  conducted. 

Under  the  oversight  of  OSD,  the  lead 
Service  will  conduct  a  JFS  to  expand  and  refine 
the  candidate,  assess  the  need  and  feasibility  of 
executing  a  JT&E,  and  prepare  a 
recommendation  to  the  SAC  and  the  D,T,SE&E 
relative  to  potential  courses  of  action.  The 
designated  participating  Services  will  provide 
resources  as  stipulated  in  the  JFS  directive. 

The  JT&E  Technical  Advisory  Board 
(TAB)  is  an  organization  of  senior  civilian 
scientists  and  engineers  from  OSD  and  the 
Services  that  provides  advice  to  the  JT&E  PC, 
SAC,  OSD  sponsors,  and  the  FSD  on  technical 
issues  related  to  the  JT&E  program.  DoD 
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5000.3-M-4  outlines  the  membership,  duties, 
and  responsibilities  of  the  TAB.  The  TAB 
chairman  also  serves  as  the  technical  advisor  to 
and  is  a  nonvoting  member  of  the  SAC.  The 
FSD  will  formally  present  test  concepts,  test 
designs,  and  study  results  to  the  TAB  for 
technical  review,  comments,  and 
recommendations.  It  should  be  noted  that  JT&E 
PC  members,  in  addition  to  representing  their 
organization/Service  during  the  nomination 
process,  frequently  are  called  on  to  act  as  the 
T&E  Liaison  for  the  members.  When  the 
DDT&E  is  satisfied  that  the  JFS  is  accurate  and 
complete,  the  FSD  will  present  the  JFS  results  to 
the  SAC.  The  SAC  will  consider  the  JFS  final 
report  and  recommend  to  the  D,T,SE&E  whether 
or  not  the  proposed  JT&E  should  be  chartered 
for  execution. 


If  the  JFS  involves  issues  that  pertain  to 
policy,  doctrine,  or  tactics,  the  FSD  should 
consider  the  option  of  establishing  a  General 
Officer  Steering  Committee  (GOSC).  The 
GOSC  would  consist  of  flag  officers  (1-2  star 
level)  from  the  Services  who  would  advise  the 
FSD  on  Service  policy,  doctrine,  tactics,  and 
roles/missions  related  to  the  JT&E.  The 
organization  and  roles  of  the  GOSC  members 
would  be  dictated  by  the  particular  objectives  of 
the  JT&E.  If  established,  the  FSD  should 
include  the  GOSC  in  the  review  cycle  of  JFS 
concepts,  products,  and  documents. 
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CHAPTER  2 

ESTABLISH  AND  ORGANIZE  THE  JFS 


A.  INTRODUCTION 

This  chapter  explains  the  relationship  of  the 
JFS  to  the  JT&E  and  describes  those  actions  that 
must  be  accomplished  upon  activation  to  obtain 
facilities,  personnel,  and  administrative  support. 
The  chapter  also  addresses  the  functional  and 
organizational  requirements  that  should  be 
considered  in  structuring  the  JFS. 

B.  INITIAL  ASSISTANCE 

One  of  the  first  tasks  of  the  newly 
appointed  FSD  will  be  to  contact  the  lead 
Service,  participating  Services,  OSD,  and  as  an 
option  the  Defense  Evaluation  Support  Activity 
(DESA)  JT&E  POCs  to  discuss,  in  detail, 
requirements,  available  support,  reporting,  and 
expectations.  Annex  C  is  a  list  of  Service 
headquarters,  operational  test  agencies  (OTA), 
OSD,  and  DESA  JT&E  POCs.  JFS  support  is 
the  responsibility  of  the  lead  Service.  The  FSD 
should  establish  liaison  with  the  POCs  at  both  the 
Service  headquarters  and  OTA  levels  because 
both  have  an  interest  and  involvement  in  the 
potential  JT&E. 

When  a  JFS  is  directed,  the  lead  Service 
headquarters  will  issue  an  order,  letter,  or 
directive  that  designates  a  subordinate  command 
or  agency  to  function  as  their  support  agent  for 
the  conduct  of  the  JFS.  Likewise,  the 
headquarters  of  the  participating  Services  will 
designate  a  subordinate  command  or  agency  to 
function  as  their  JFS  support  agent. 


The  nomination  process  itself  involves  a 
number  of  planning  and  coordination  activities 
which  relate  directly  to  JFS  requirements.  All 
documents  supporting  these  activities  should  be 
obtained  from  the  lead  Service  headquarters 
POC. 

As  a  minimum,  the  following  documents 
should  be  obtained. 

JT&E  nomination  documents  contain 
a  description  of  the  proposed  JT&E  in  terms  of 
purpose  and  expected  results,  concepts,  issues, 
objectives,  problems,  utility,  and  impact  on  joint 
effectiveness.  These  documents  also  identify 
anticipated  users  of  the  results  and  provide  a 
proposed  program  schedule/milestones;  the 
identification  of  unique  hardware,  software, 
environment,  personnel,  and  instrumentation 
requirements;  an  outline  of  a  test  concept;  a 
proposed  JFS  plan  of  action  and  milestones;  an 
estimate  of  funds  and  resources  required;  known 
or  projected  shortfalls;  and  CINCs/agencies/ 
organizations  concurring  in  the  nomination. 
While  some  nomination  documentation  is  more 
complete  than  others,  it  should  still  serve  as  the 
starting  point  for  every  JFS  because  it  provides 
the  FSD  with  an  understanding  of  the  issues  or 
proposed  concepts  as  seen  through  the  eyes  of 
the  nominating  command  or  agency. 

The  JT&E  PC  reviews  the  nominations; 
if  applicable,  recommends  combining  with  other 
nominations;  identifies  opportunities  for 
incorporation  into  activities  already  scheduled  in 
conjunction  with  joint  or  Service  exercises; 
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determines  multi-Service  participation  and  the 
ability  of  the  Services  to  provide  the  necessary 
manpower  and  facilities;  considers  the  utility  of 
the  results;  and  estimates  the  impact  of  these 
results  on  joint  capabilities.  The  JT&E  PC 
minutes  also  provide  initial  inter-Service 
coordination  and  identifies  the  major  participants 
and  users  of  test  results  who  must  be  included  in 
the  coordination  of  JFS-produced  documents  and 
products. 

The  SAC  reviews  the  JT&E  PC 
recommendations  and  recommends  which 
nominations  should  become  candidate  JFSs.  The 
SAC  minutes  also  prioritize  those  candidates  that 
are  recommended  for  approval  and  suggests 
which  require  a  JFS  and  which  should  proceed 
directly  to  a  JT&E.  A  SAC  recommendation  for 
approval  is  Service  and  Joint  Chief  of  Staff 
(JCS)  concurrence  in  the  selected  candidate(s) 
concepts  or  issues  and  its  (their)  definitive 
objectives.  It  is  also  an  initial  commitment  that 
Service  support  will  be  available.  It  is  very 
important  that  concepts,  issues, 
objectives,  personnel,  equipment,  and 
cost  be  solidified  early  in  the  JFS.  The 
SAC  report  (Service  and  Joint  Staff 
concurrence )  is  one  of  the  key 
documents  to  be  used  to  achieve  this 
end.  Additionally,  the  SAC  priority  listing  is 
indicative  of  JT&E  importance  to  DoD  in  the 
event  that  budget  reductions  occur  and  alternative 
considerations  are  required. 

JFS  directive  authorizes  the  execution  of 
the  JFS,  designates  the  lead  and  participating 
Services,  and  approves  the  appointment  of  the 
FSD.  This  directive  contains  the  concepts  or 
issues  and  objectives  and  provides  for  JT&E 
program-unique  funding  for  the  JFS.  Since  it 


represents  OSD  concurrence,  it  should  be  used  in 
conjunction  with  the  SAC  report  to  solidify 
JT&E  concepts  or  issues  and  objectives.  This  is 
not  to  say  that  they  cannot  be  revised  or  rescoped 
as  the  JFS  concept  matures.  Revisions  and 
changes  should  be  considered  when  necessary. 
If  significant  revisions  or  changes  are  needed, 
they  should  be  made  with  the  DDT&E 
concurrence. 

Service  directives  and  instructions 
relating  to  JFS  support  are  written  by  the 
headquarters  staffs  of  the  lead  and  participating 
Services  to  subordinate  commands  or  agencies 
outlining  responsibilities  and  priorities  for 
providing  support  for  the  JFS.  These 
directives/instructions  provide  the  FSD  and 
Service  deputies  with  the  authority  to  coordinate 
directly  with  supporting  organizations  to  obtain 
facilities,  administrative  support,  and  staff 
personnel.  These  directives  also  provide  the 
designated  support  agents  with  the  authority  to 
provide  the  required  support. 

Archive  documents  produced  by 
completed  or  ongoing  joint  tests  are  maintained  in 
the  AFOTEC  OT&E  Data  Branch  Library.  This 
library  contains  numerous  charters,  directives, 
Program  Test  Designs  (PTDs),  Program  Test 
Plans  (PTPs),  final  reports,  and  management 
reports.  While  some  of  these  documents  may  not 
directly  relate  to  the  JFS,  many  JT&E  challenges 
are  systemic  to  the  process  and  have  been 
previously  faced  and  documented  by  other  JFSs 
and  JT&Es.  Of  specific  interest  should  be  the 
management  reports,  which  contain  the  lessons 
learned  from  previous  JFSs  and  JT&Es  along 
with  the  director's  recommended 
improvement/corrective  actions.  This  material  is 
available  to  all  FSDs  and  their  staffs.  The  FSD 
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should  visit  the  AFOTEC  library  at  Kirtland  AFB 
to  review  and  determine  which  of  the  available 
documents  may  be  applicable.  Request  for 
documents  should  be  addressed  to: 


AFOTEC/RS 

Directorate  of  Research  Service 
8500  Gibson  Blvd.  SE 
Kirtland  AFB,  NM  87117-5558 
Telephone:  (505)  846-2574/DSN  246-2574 


C.  IMMEDIATE  REQUIREMENTS 

When  a  JFS  is  activated,  the  FSD  will  be 
faced  with  a  myriad  of  requirements  that  include 
establishing  and  organizing  the  JFS,  conducting 
literature  searches,  obtaining  documents  and 
directives,  establishing  liaison  with  POCs,  etc. 
Immediate  requirements  will  center  on: 

•  Obtaining  an  adequate  JFS  Headquarters 
site. 

•  Obtaining  qualified  staff  and  support 
personnel  in  a  timely  manner. 

•  Obtaining  support  resources  (e.g., 
communications  and  computer  equipment). 

•  Refining  the  JFS  budget  and  securing  the 
required  funds. 

•  Organizing  the  JFS  and  developing  a  work 
breakdown  structure  (WBS)  and  schedule. 

•  Establishing  support  and  coordination 
POCs  among  the  Services,  Joint  Staff,  and 
OSD. 

•  Providing  for  security  of  JFS  actions, 
documents,  and  records. 

A  key  to  a  good  JFS  is  a  lean 
organization  staffed  with  fully  qualified  and 
experienced  personnel  from  all  the  Services 
who  maintain  continuous  communications  with 
the  Services.  The  FSD  should  have  been 


identified  during  JT&E  PC  coordination.  In  the 
case  of  a  Service  nomination,  the  nominating 
Service  will  be  assigned  as  the  lead  Service  and 
the  nominating  organization  within  that  Service 
will  most  probably  be  directed  to  function  as  the 
JFS  support  agent  (the  nominating  organization 
has  interest  and  involvement).  Ideally,  the 
proposed  FSD  should  have  been  involved  in  the 
JT&E  nomination  process  and  be  immediately 
available  to  initiate  JFS  actions.  The  designated 
participating  Services  will  be  aware  of  support 
personnel  requirements  (Services  agreed  during 
JT&E  PC  coordination)  and  should  provide  the 
Service  deputies  and  support  personnel  on  a 
temporary  duty  (TDY)  basis  as  soon  as  they  are 
requested. 

Initiation  of  the  JFS  activities  should  begin 
almost  immediately  after  the  D,T,SE&E  issues 
the  JFS  directive.  The  lead  Service  support 
agents  will  use  their  own  facilities,  personnel, 
and  resources  to  conduct  the  JFS.  OSD  will 
provide  funds  to  pay  for  costs  that  are  unique  to 
the  needs  of  the  JFS.  DESA  is  available  to 
provide  management  (contract,  safety,  security, 
resource,  environmental,  logistical,  and  fiscal) 
and  technical  (engineering,  analysis,  and 
instrumentation)  support.  In  order  to  obtain 
DESA  support,  the  FSD  should  contact  the 
DESA  JT&E  POC  to  develop  a  Program 
Initiation  Document  (PID)  between  the  FSD  and 
DESA. 

In  the  case  of  CINC/OSD/Joint  Staff 
nominations,  the  JT&E  PC  will  recommend 
which  of  the  Services  should  function  as  lead  and 
which  should  provide  support.  From  that  point 
on,  the  procedures  for  assigning  JFS 
responsibilities  are  the  same  as  described  above. 
If  the  organization  responsible  for  conducting  the 
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JFS  is  not  the  nominating  organization  or 
agency,  some  delay  may  occur  as  the  lead 
Service  identifies  and  acquires  the  required 
facilities  and  resources  and  assigns  the  FSD. 

The  FSD  is  directly  responsible  to  the 
DDT&E  for  the  organization,  management,  and 
execution  of  the  JFS.  The  FSD  is  authorized  by 
the  D,T,SE&E  to  establish  direct 
communications  with  Service  or  DoD  agencies  to 
obtain  assistance  or  support  required  for  the  JFS. 
One  of  the  major  aspects  of  the  JT&E  program 
that  is  most  frequently  underestimated  is  that  the 
FSD  will  be  required  in  a  relatively  short  time 
frame  to  establish  a  totally  new  and  independent 
organization  with  all  its  attendant  problems.  As  a 
minimum,  initiating  the  JFS  effort  will  involve 
establishing  an  organizational  structure  and 
obtaining  adequate  facilities,  qualified  staffing, 
and  support  resources. 

Facilities 

The  JFS  directive  and  Service  order,  letter, 
or  directive  that  implements  the  JFS  will  stipulate 
where  the  JFS  will  be  located  and  who  is 
responsible  for  providing  the  required  facilities, 
resources,  and  support.  Facilities,  resources, 
and  support  include  such  things  as  heat,  light, 
furniture,  safes,  administrative  equipment, 
computers,  and  access  to  network  bulletin  boards 
and  on-line  services  (e.g.,  MILNET/TECNET). 
Provision  of  this  support  is  the 
responsibility  of  the  lead  Service.  Some 
installations  have  facilities  established  and 
designated  to  support  short  term  projects  such  as 
JFSs.  On  other  installations,  the  FSD  and  the 
designated  support  agent  POC  must  negotiate 
with  the  host  installation  to  obtain  the  required 
facilities.  Assuming  the  JFS  will  consist  of  10- 


12  personnel,  the  FSD  should  negotiate  to 
acquire  1500-2500  square  feet  of  usable  office 
floor  space,  preferably  at  a  single  location. 
While  most  of  the  space  will  be  used  for  offices, 
one  conference  area  of  at  least  500  square  feet 
should  be  available  for  use  also  as  a  collective 
WORK  area. 

In  most  JFSs,  some  of  the  work  will 
involve  classified  materials.  Controlled  access  to 
the  JFS  facility  and  the  storage  of  classified 
materials  must  be  considered.  If  the  JFS 
involves  information  and  discussions  that  require 
a  Special  Compartmented  Information  Facility 
(SCIF),  these  special  requirements  must  be 
addressed  through  the  designated  support  agent 
POC. 

Staff 

Designation  as  the  lead  Service  carries  the 
responsibility  of  supporting  the  JFS  with  the 
director,  one  of  the  deputies,  and  the  majority  of 
Service  personnel;  and  providing  administrative 
services  and  assistance  in  the  areas  of  secretarial 
support,  travel,  contracting,  personnel, 
administration,  comptroller,  supply,  and  logistics 
(see  DoD  5000.3-M-4 ).  Participating  Services 
will  provide  deputies  as  stipulated  in  the  JFS 
directive.  DoD  5000.3-M-4  further  states  that  the 
Services  will  provide  fully  qualified  personnel 
(both  military  and  civilian)  to  staff  the  JFS.  One 
of  the  FSD's  initial  tasks  will  be  to  determine 
staff  requirements  in  terms  of  numbers,  rank, 
and  qualifications  and  submit  these  requirements 
to  the  Service  headquarters  POCs  for  action  by 
the  Service  personnel  systems.  If  the  FSD  was 
involved  in  the  JT&E  PC,  JFS  staff  requirements 
should  have  been  established  and  coordinated 
with  both  the  lead  and  participating  Services 
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during  the  JT&E  nomination  process.  Staffing 
these  positions  will  then  depend  on  the  ability  of 
Service  personnel  systems  to  respond  to  FSD 
requirements. 

While  staffing  procedures  are  being 
established,  it  is  sometimes  difficult  for  the 
Services  to  provide  qualified  personnel  in  a 
timely  manner.  Available  personnel,  or 
personnel  who  happen  to  be  on  site,  are 
sometimes  assigned  to  these  positions.  Others 
have  been  assigned  as  an  additional  duty.  The 
FSD  should  insist  on  the  assignment  of  fully 
qualified  full-time  personnel,  and  all  should  be 
required  to  be  located  at  the  JFS  location.  This 
could  be  accomplished  through  the  use  of 
Service-sponsored,  long-term  training  opportuni¬ 
ties.  Service  POCs  should  be  queried  as  to  the 
availability  of  this  option.  While  this  might  result 
in  the  obtaining  of  qualified  personnel  for  the 
duration  of  the  JFS,  it  should  be  noted  that  these 
opportunities  normally  require  a 
nomination/selection  process  that  can  be 
somewhat  time  consuming.  The  FSD  should 
also  insist  that  JFS  personnel  be  available  for 
transition  to  the  JTF  should  the  JTF  be  chartered. 

The  conduct  of  the  JFS  will  require 
personnel  with  varied  operational  and  technical 
skills.  Past  JFSs  have  indicated  that  a  full-time 
core  of  personnel  with  technical  expertise  in  the 
areas  of  military  operations,  test  and  evaluation, 
analysis,  and  engineering  will  probably  be 
required.  Additional  personnel  with  skills  in 
resource  management,  logistics,  transportation, 
etc.  may  be  required  on  a  part-time  basis.  Some 
considerations  for  staffing  are: 

Feasibility  Study  Director  should  be 
an  0-6  (possibly  a  GS-15)  with  test  and 


evaluation  experience  and  a  sound  technical 
background.  While  joint  experience  is  desired, 
command  experience  is  necessary  due  to  the 
problems  with  establishing  a  new  command. 
The  requirement  to  brief  flag  levels  also  dictates 
an  0-6.  Senior  0-5s  are  acceptable  if  they  have 
promotion  potential  and  the  required  T&E  and 
technical  experience.  It  is  expected  that  the  FSD 
will  transition  and  become  the  JTD. 

Military  operations  personnel  who 
possess  knowledge  and  experience  in  the 
operational  aspects  of  the  JFS  subject  matter  will 
be  required.  These  individuals  should 
understand  the  concepts  or  issues  from  a  joint 
perspective  or  at  least  from  the  perspective  of 
their  respective  Services.  They  should  also  be 
versed  in  their  Service's  mission,  doctrine, 
tactics,  procedures,  and  systems  that  are 
applicable  to  the  nominated  concepts  or  issues. 
They  should  also  be  familiar  with  Service  and 
Joint  Staff  exercise  and  training  activities 
applicable  to  the  JFS  operational  issues. 

Test  and  evaluation  expertise  is  an 
absolute  necessity  to  the  successful  conduct  of 
the  JFS.  Personnel  must  be  experienced  in 
concept  development,  analysis,  planning, 
execution,  and  reporting  of  testing  activities. 
This  experience  should  include  the  identification 
of  test  events  and  locations;  the  development  and 
management  of  test  schedules  and  budgets;  the 
identification,  acquisition,  and  management  of 
test  resources;  and  data  collection  and 
management.  These  personnel  should  also  be 
able  to  plan  for  test  analysis,  develop  program 
test  designs,  develop  test  plans  and  procedures, 
and  write  reports. 
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Analysis  personnel  with  experience  in 
operations  research,  systems  analysis,  or  related 
scientific  disciplines  and  experience  in  the 
application  of  these  skills  to  joint  testing  will  be 
required.  This  experience  should  include 
developing  test  designs,  defining  evaluation 
criteria,  identifying  data  requirements,  using 
automated  and  manual  analysis  tools  and 
techniques,  and  applying  modeling  and 
simulation  techniques  to  satisfy  analysis 
requirements. 

Engineering  personnel  with  testing 
experience  may  be  required.  This  expertise  may 
include  the  employment  of  test  systems  and  the 
integration  of  test  instrumentation  system  and 
associated  interface  equipment.  Engineering 
experience  is  required  to  identify  the  technical 
parameters  associated  with  obtaining  data  to 
satisfy  the  test  issues/objectives  and  the  types  of 
instrumentation  systems  that  will  be  required  to 
collect  the  data. 

Contractor  personnel  with  a  variety  of 
JT&E  background  and  experience  may  be 
required.  The  time  allowed  for  a  JFS  precludes 
delays  in  obtaining  qualified  personnel,  the 
training  of  unqualified  personnel,  or  the 
inefficiencies  inherent  in  the  use  of  part-time  or 
additional  duty  personnel.  If  the  Service 
personnel  systems  cannot  accommodate  the  JFS 
personnel  requirements  in  a  timely  manner,  the 
FSD  must  consider  the  use  of  qualified  contractor 
personnel  to  support  the  JFS.  Contractor 
support  can  be  obtained  by  either  modifying 
existing  contracts  or  by  using  a  DESA  support 
contract.  Either  will  involve  the  development, 
coordination,  and  approval  of  a  task  statement 
specifically  addressing  the  JFS  requirements. 
Managerial  and  technical  support  or  assistance  in 


contract  administration  is  available  from  DESA. 
The  FSD  should  contact  the  DESA  JT&E  POC 
for  such  assistance. 

Communications  and  Automated  Data 
Processing  (ADP)  Support 

The  provision  of  adequate  communications 
and  ADP  equipment  is  the  responsibility  of  the 
support  agent  designated  by  the  lead  Service. 
The  identification  of  equipment  required  and  the 
requisition  of  this  equipment  are  the 
responsibility  of  the  FSD.  Most  organizations 
have  some  communications  and  ADP  equipment 
on  hand.  The  FSD  should  coordinate  with  the 
designated  support  agent  to  determine  if 
equipment  is  available.  DESA  may  be  a  source  if 
the  lead  service  support  agent  is  unable  to 
provide  the  required  ADP  support.  DESA 
maintains  an  inventory  of  residual 
communications  and  ADP  equipment  from  past 
JT&Es.  Current  JFS  and  JT&Es  have  priority  of 
use  of  this  equipment. 

JFS  communications  and  ADP  capabilities 
should  include  at  least  some  computers  of  the 
486  class  or  higher  to  accommodate  databases, 
planning,  programming,  analysis,  and  graphics 
requirements.  FAX/modem  capabilities  are 
essential.  Communications  and  ADP  equipment 
that  are  not  available  from  on-hand  stocks  may 
take  four  to  six  weeks  to  acquire,  and  this  could 
impact  the  timely  completion  of  the  JFS.  In  any 
case,  core  communications  and  ADP 
requirements  and  the  initiation  of  actions  to  lease 
or  acquire  this  equipment  require  immediate 
action  by  the  FSD.  Leasing  equipment  is  rarely 
cost  effective  for  anything  over  a  three-month 
period.  The  FSD  should  also  consider  follow-on 
JTF  requirements  in  any  decision  to  purchase 
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communications  and  ADP  equipment;  in  all 
likelyhood,  the  equipment  will  be  transferred  to 
the  JTF  if  it  is  chartered. 

Funds 

The  lead  Service  will  fund  those  costs 
associated  with  the  facilities,  assigned  and 
supporting  civilian  personal,  and  all 
administrative  and  routine  logistical  support. 
This  is  true  even  if  the  JFS  is  conducted  at  an 
industrially  funded  location.  It  should  be  noted 
that  the  cost  to  a  lead  Service  will  increase 
significantly  for  industrial  funded  locations. 
The  participating  Services  will  provide  salaries 
for  their  JFS  staff  personnel. 

The  DDT&E  disperses  PE  0605804D  funds 
in  accordance  with  DoD  71 10.1  -M,  the  Budget 
Guidance  Manual  ( Chapter  251 ),  to  pay  for  costs 
that  are  unique  to  a  JFS.  Unique  costs 
include  travel,  per  diem,  and  dedicated 
contractor  efforts  expended  in  support  of 
the  JFS.  OSD  will  initially  provide  these  funds 
based  on  the  resource  estimates  that  were 
established  in  the  nomination  report  to  the  SAC. 
Although  this  budget  estimate  was  developed  by 
the  nominating  sponsor  and  briefed  as  part  of  the 
JT&E  PC  and  SAC  decision  processes,  it  should 
be  revised  by  the  FSD  based  upon  updated  JFS 
information.  The  revised  budget  estimate  should 
be  forwarded  to  DDT&E  within  30  days 
following  issuance  of  the  JFS  directive.  The 
revised  budget  will  include  up-dated  estimates 
separated  into  the  major  categories  of  travel,  per 
diem,  and  contractor  costs.  The  budget  revision 
will  be  reviewed  by  the  DDT&E  and,  after 
considering  the  financial  impact  and  discussions 
with  the  FSD,  revised  funding  will  be  set  aside 
for  the  conduct  of  the  JFS.  DDT&E  will  then 


provide  funds,  either  in  full  or  incrementally,  for 
the  duration  of  the  JFS.  The  FSD  will  provide 
periodic  or  as  required  financial  status  reports  to 
include  commitments  and  expenditures  by 
category  and  available  funding  balance.  Further 
revision  of  the  budget  estimate  will  be  as 
negotiated  between  the  FSD  and  DDT&E. 

The  FSD  is  responsible  for  operating 
within  the  bounds  of  the  approved  revised 
budget.  If  there  are  expenses  that  are  not  in  the 
revised  budget,  the  FSD  will  obtain  the 
concurrence  of  DDT&E  prior  to  the  commitment 
of  funds.  Service  approval  will  be  obtained  for 
any  expenses  that  are  funded  by  the  Services.  If 
discussions  arise  relative  to  the  funding  of  any 
JFS  activity  or  support,  the  FSD  should 
immediately  contact  the  applicable  JT&E  POC  for 
clarification  and  resolution. 

The  FSD  should  establish  a  relationship 
with  the  host  installation  comptroller  early  in  the 
program  to  identify  support  needs  for 
reimbursable  programs  and  establish  applicable 
fiscal  accounts.  The  FSD  should  advise  DDT&E 
and  the  appropriate  Service  agencies  of  these 
accounts  so  that  OSD  and  Service  fund  transfers 
can  be  directed  accordingly. 

Managers  at  all  levels  in  DoD  track  funds 
by  the  three  phases  in  the  accounting  cycle: 
commitment,  obligation,  and  expenditure.  The 
FSD  and  JFS  financial  manager  must,  therefore, 
understand  the  phases,  the  impact  of  financial 
related  actions  taken  upon  them,  and  the 
documentation  required  to  execute  the  financial 
programs  and  be  able  to  report  on  the  status  of 
individual  JFS  actions  in  terms  of  the  accounting 
cycle.  The  following  is  a  brief  overview  of  the 
three  phases  of  fiscal  accountability. 
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Commitment  is  an  administrative 
reservation  of  funds  evidencing  an  intent  to  incur 
an  obligation.  Commitments  are  based  on  firm 
requisitions,  purchase  requests,  or  allied 
documents  that  will  require  actual  contracting 
actions,  or  other  authorized  written  evidence  that 
indicates  intention  to  incur  obligations. 
Commitments  involve  a  certification  of  funds 
availability  that  attests  that  funds  are  currently 
available  to  cover  potential  obligations  and  that 
these  obligations  are  proper  and  valid  charges  to 
the  funds  cited.  Commitments  are  made  against 
funds  available  for  the  period  in  which  the 
obligation  is  to  be  incurred.  Decommitments  of 
funds  should  be  accomplished  to  increase 
available  funds  anytime  a  requirement  no  longer 
exists,  funds  cannot  be  obligated  within  the 
prescribed  time  frame  (fiscal  year  for  Service 
funds  and  two  years  for  OSD  PE  06058D4D 
funds),  or  a  credit  commitment  document  is 
received. 

Obligation  is  a  documented  transaction 
that  constitutes  a  legal  requirement  for  furnishing 
goods  or  services  and  a  corresponding 
requirement  to  pay  for  those  items  upon 
completion  or  delivery.  An  obligation  is  the 
amount  of  an  order  placed,  a  contract  awarded, 
services  received,  or  similar  transaction  during  a 
given  period  that  will  require  payments  during 
either  the  same  or  future  periods.  Obligations 
should  be  recorded  only  when  documentary 
evidence  supports  the  amount  of  the  obligation. 
All  obligation  documents  should  be  sent 
promptly  to  the  servicing  accounting  and  finance 
officer  for  recording  once  an  obligation  has  been 
incurred.  An  annual  appropriation  is  available  for 
obligation  only  during  the  fiscal  year  specified  in 
the  appropriations  act.  In  order  to  obligate  an 
annual  appropriation,  the  contract  must  be  made 


within  the  fiscal  year  to  be  charged  against,  the 
supplies  or  services  must  serve  a  bona  fide  need 
of  that  fiscal  year,  and  the  obligation  cannot 
exceed  the  available  allotment.  Obligations  may 
be  incurred  at  any  time  during  the  period  of  a 
multiple  year  appropriation  if  the  related  action 
concerns  an  item  authorized  in  the  appropriation 
act  and  the  obligation  does  not  exceed  the 
approved  allotment. 

Expenditure  occurs  when  an  invoice  or 
similar  document  has  been  received  and  payment 
has  been  made  against  it  to  satisfy  an  obligation. 
Accrued  expenditures  are  charges  to  an 
appropriation  during  a  period  that  requires  funds 
either  be  dispersed  or  set  aside  for  payment. 
Expenditures  accrue  regardless  of  when  cash 
payments  are  made  and  are  recorded  based  on 
documentary  evidence  including  bills  of  lading, 
contractor  invoices,  and  obligating  documents.  A 
recordable  accrued  expenditure  includes  the 
amount  of  work  accomplished  in  spite  of  the 
stage  of  completion,  shipping  status,  or  lack  of 
acceptance  by  the  Government. 

The  difference  between  the  funding 
received  by  fiscal  year  and  that  portion  which  has 
been  set  aside  as  committed,  obligated  or 
expended  is  the  available  balance  left  to  conduct 
operations  during  the  lifetime  of  the  funds.  It  is 
important  to  remember  that  the  combined  total  of 
the  available  balance,  prior  expenditures,  current 
obligations,  and  commitments  can  not  exceed  the 
authorized  funding  by  source  and  appropriation. 
More  importantly,  it  is  a  violation  of  public  law 
(Anti-Deficiency  Act)  if  expenditures  and  legal 
obligations  exceed  the  available  funding  of  each 
separate  funding  document.  Additionally,  funds 
can  not  be  commingled,  i.e.  OSD  an,d  Service 
funds  accounted  for  as  one. 
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Figure  2-1.  Example  JFS  Organization  Structure 


Organization 

While  most  organizational  structures  can  be 
tailored  to  accommodate  a  JFS,  a  simple  structure 
that  is  based  on  JFS  tasks  or  products  will 
probably  work  best.  The  selected  organizational 
structure  should  provide  a  clear  chain  of 
responsibility  and  accountability  for  each 
function,  activity,  task,  or  product.  Figure  2-1 
is  an  example  of  such  an  organizational  structure. 
The  organizational  structure  should  be  tailored  to 
take  into  account  the  unique  nature  and  technical 
complexity  of  the  JFS. 

Because  of  the  differences  in  the  Services 
and  the  varied  operational  and  technical 
backgrounds  of  the  assembled  staff,  intolerance 
to  broad  ranges  of  ideas  and  opinions  often 
develops  within  the  JFS.  It  is  for  this  reason  that 
someone  within  the  JFS  should  function  as  the 
focal  point  for  resolving  differences  in  technical 
ideas  and  opinions  and  maintain  the  focus  of  JFS 


efforts  on  the  directed  mission.  This  can  best  be 
accomplished  by  the  designation  of  a  JFS 
technical  advisor.  In  most  cases  the  FSD  will 
have  the  technical  competence  to  function  as  the 
technical  advisor.  The  FSD  must  recognize, 
however,  that  immersion  in  the  technical  aspects 
of  the  JFS  sometimes  is  at  the  expense  of 
management  functions.  In  this  case,  the  FSD 
may  want  to  consider  the  establishment  of  a 
separate  technical  advisor  position.  It  may  be 
difficult  to  fill  this  position  with  Service  or 
civilian  personnel  due  to  the  JFS  duration  of  less 
than  a  year  and  the  uncertainty  of  chartering  the 
follow-on  JTF.  The  JFS  should  consider  filling 
this  position  by  competent  contractor  support, 
preferably  from  a  contractor  other  than  the 
primary  support  agent. 

The  FSD  reports  to  the  DDT&E  and  will  be 
assisted  by  the  Service  deputies.  The  Service 
deputies  should  be  responsible  for  major 
functions  or  divisions  in  the  JFS  organizational 
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structure.  The  specific  responsibilities  of  the 
FSD  and  Service  deputies  are: 

FSD 

•  Establish,  organize,  and  supervise  the  JFS 
team. 

•  Lead,  manage,  and  supervise  all  facets  of 
the  JFS. 

•  Ensure  the  JFS  is  conducted  to  accomplish 
its  mission  on  schedule  and  within  budget. 

•  Develop  a  JFS  WBS  and  schedule. 

•  Maintain  contact  with  applicable  OSD/Joint 
Staff/Service  committees,  agencies,  and 
support  entities. 

•  Develop,  maintain,  and  update 
requirements  for  OSD  funding  and  Service 
support  and  submit  these  requirements  to 
the  DDT&E  and  the  Services  as 
appropriate. 

•  Control  funds  designated  for  JFS  activities 
and  account  to  the  DDT&E  for  their  use. 

•  Monitor  Service  expenditures  related  to  the 
JFS. 

•  Prepare  and  submit  reports  as  appropriate 
(in  process  review,  final  report,  APA,  JFS 
management  report). 

Service  Deputies 

•  Ensure  that  their  own  Service  concerns  are 
adequately  addressed  by  the  JFS. 

•  Identify  Service  resource  and  personnel 
expertise  required  for  execution  of  the  JFS 
and  ensure  their  availability. 

•  Coordinate  within  own  Service  to  obtain 
required  information  and  resources. 

•  Provide  advice  and  expertise  to  the  JFS  on 
Service  issues  and  areas  of  interest. 


•  Provide  progress  and  significant  action 
reports  to  their  own  Service. 

•  Obtain  coordination  of  own  Service  relative 
to  the  feasibility  and  value  of  JFS  actions, 
conclusions,  recommendations,  and 
products. 

Security 

Most  JFSs  will  involve  access  to  or  the 
production  of  sensitive  or  classified  materials.  If 
so,  the  FSD  must  establish  procedures  for  the 
complete  security  of  JFS  functions.  These 
procedures  should  include  considerations  for  the 
establishment  of  controlled  access  areas;  the 
positive  identification  and  control  of  personnel  in 
the  controlled  access  areas;  the  receipt,  storage, 
transmittal,  reproduction,  and  destruction  of 
classified  documents;  the  control  of  classified 
containers  and  the  container  combinations; 
communications  and  ADP  security;  personnel 
security  clearances;  and  facility  security  checks. 
Service  security  manuals  and  DoD  5220.22-M 
(Industrial  Security  Manual  for  Safeguarding 
Classified  Information)  provide  detailed 
information  relative  to  the  Government's 
Information  Security  Program  and  minimum 
requirements  for  safeguarding  classified 
information. 

If  the  JFS  requires  access  to  Sensitive 
Compartmented  Information  (SCI),  the  FSD 
must  ensure  that  billets  are  available  when  needed 
through  an  existing  Special  Security  Office 
(SSO)  or  DESA  to  support  this  requirement. 
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Security  Classification  Guide  (SCG) 

Another  consideration  associated  with 
classified  or  sensitive  projects  is  that  of  marking 
newly  created  documents.  The  classification  of 
documents  or  portions  of  documents  must  be 
based  on  the  information  they  contain  or  reveal. 
JFS  classification  authority  will  normally  be 
derived  from  the  original  documents.  If  the  JFS 
involves  the  development  of  classified  material, 
contact  the  OSD  JT&E  Coordinator  for  assistance 
and  information  relative  to  establishment  of 


original  classification  authority.  OSD  security 
provides  original  classification  authority  in  an 
annual  memorandum  issued  each  fall. 

Each  Service  has  its  own  unique 
requirements  for  Security  Classification  Guides. 
The  JFS  should  use  service-unique  and 
established  joint  procedures  when  possible.  If  a 
SCG  is  needed,  the  FSD  should  contact  the  OSD 
JT&E  Coordinator  to  obtain  assistance  and 
support  for  its  development.  An  outline  for  a 
SCG  that  can  be  adapted  to  meet  the  needs  of  the 
JT&E  is  in  Annex  L. 
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CHAPTER  3 

JOINT  FEASIBILITY  STUDY  EXECUTION 


A.  INTRODUCTION 

This  chapter  provides  an  overview  of  those 
activities  and  actions  required  to  conduct  a  JFS, 
develop  a  test  concept  and  APA,  prepare  the  JFS 
final  report,  and  provide  recommendations 
relative  to  a  follow-on  JT&E  conducted  by  a 
JTF.  Part  One  addresses  JFS  execution.  Part 
Two  addresses  the  development  of  a  APA.  The 
FSD  must  focus  on  meeting  the  TAB  and  SAC  in 
May  and  June  with  a  fully  prepared  and  fully 
coordinated  JFS  final  report  and  APA.  Part 
Three  provides  an  overview  of  the  program 
reporting  requirements  for  the  JFS.  The  FSD 
should  not  miscalculate  the  time  and  effort 
required  to  coordinate  a  JFS  final  report.  It 
usually  takes  two  drafts  starting  in  January.  The 
first  draft  serves  to  obtain  Service  comments  and 
ideas  and  the  second  draft  becomes  the  "For 
Record"  version.  Most  of  the  time  between  the 
May  TAB  and  the  June  SAC  will  be  required  to 
polish  the  JFS  final  report  and  pre-brief  the  SAC 
members. 

It  is  important  to  remember  that  the  primary 
purpose  of  a  JFS  is  to  provide  those  involved  in 
the  review  and  decision  making  process, 
specifically  the  SAC  and  DDT&E,  with  sufficient 
information  to  make  intelligent  and  well  informed 
decisions  regarding  the  chartering  of  a  JT&E.  To 
this  end,  the  JFS  must  address,  in  sufficient 
detail,  the  answer  to  two  critical  questions: 


•Isa  JT&E  necessary  ? 
•  Is  a  JT&E  feasible  ? 


The  Milestone  I  Review  will  concentrate  on 
the  necessary  issue.  If  the  JFS  determines  that 
the  proposed  nomination  is  not  necessary,  the 
FSD  should  present  the  JFS  findings  at  that 
point.  If  the  DDT&E  agrees  that  the  JT&E  is 
not  necessary,  the  JFS  should  be  concluded,  a 
JFS  final  report  and  management  report  should 
be  prepared,  and  actions  should  be  initiated  to 
close  out  the  JFS.  The  criteria  for  determining 
necessity  are  addressed  in  Section  E  of  this 
chapter.  The  second  critical  question,  Is  a 
JT&E  feasible?,  should  be  addressed  after  the 
APA  has  been  developed  and  the  required  test 
activities  and  resources  have  been  identified  and 
quantified.  Detailed  guidance  for  answering  this 
question  is  contained  in  Section  M. 

PART  ONE 

JOINT  FEASIBILITY  STUDY 

This  part  consolidates  information, 
guidance,  recommendations,  and  POCs  to  assist 
the  FSD  and  staff  in  the  execution  of  a  JFS.  The 
following  are  discussions  based  on  techniques 
successfully  used  in  previous  JFSs.  The  FSD 
and  staff  should  consider  these  techniques  and 
tailor  them  to  the  specific  requirements  of  their 
JFS. 

B .  REVIEW  AVAILABLE 
DOCUMENTS  AND 
UNDERSTAND  THE  NOMINATED 
CONCEPTS/ISSUES 

The  JFS  is  a  discrete  but  integral  and 
sequential  part  of  the  JT&E  program. 
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Understanding  the  nomination  process,  the 
problem  or  situation  that  created  the  nomination, 
the  recommended  concepts  and  solutions,  and 
who  concurs  with  and  who  must  directly  support 
the  JT&E  is  essential. 

An  initial  step  in  conducting  the  JFS  is  to 
review  and  understand  the  documents  that  led  to 
the  nomination;  these  were  generated  during  the 
nomination  review,  coordination,  approval,  and 
chartering  process.  Section  B  of  Chapter  2 
describes  these  documents.  The  FSD  should 
contact  the  OSD  JT&E  Coordinator  at  the  earliest 
opportunity  for  assistance  in  obtaining 
documents  the  JFS  staff  may  not  have.  The  OSD 
JT&E  Coordinator  can  also  provide  assistance  in 
obtaining  current  copies  of  the  JT&E  Procedures 
Manual,  the  JFS  and  JT&E  Handbooks,  Service 
JT&E  support  directives,  and  the  JT&E  support 
Memorandum  of  Agreement  (MOA).  Further,  the 
OSD  JT&E  Coordinator  can  assist  in  determining 
what  documents  in  the  JT&E  library  may  be  of 
interest  to  the  FSD  and  assist  in  obtaining  them. 

The  JFS  staff  must  become  knowledgeable 
of  the  nominated  concepts,  issues,  and 
objectives,  to  include  background  information, 
because  they  provide  real  world  dimension  to  the 
situation  or  problem.  Nominations  are  frequently 
manifestations  of  localized  situations.  Thus,  the 
FSD  and  staff  must  quickly  become  experts  on 
the  proposed  JT&E  while  retaining  an  unbiased 
and  joint  focus  on  the  situation,  problems, 
policies,  directives,  regulations,  and  doctrine 
associated  with  the  nomination. 

The  FSD  should  remember  that  the  purpose 
of  the  JFS  is  to  address  a  concept  or  an  issue 
whose  resolution  is  unknown  at  the  start.  The 


JFS  must  be  an  objective  evaluation  of 
the  concept  or  issue,  and  the  director 
and  staff  should  not  become  advocates 
of  the  concept  or  issue  solution.  Such 
advocacy  may  either  preclude  critical  analysis  or 
invite  improper  balance  (desirable  features 
investigated  while  possible  shortcomings 
ignored)  by  the  JFS  staff. 

C .  DEVELOP  THE  JFS  WORK 

BREAKDOWN  STRUCTURE 

AND  SCHEDULE 

In  addition  to  the  revised  budget  estimate 
described  in  Chapter  2,  the  FSD  should  refine  the 
JFS  schedule  that  was  developed  during  the 
nomination  process.  The  FSD  should  develop  a 
WBS  and  schedule  that  are  based  upon  updated 
projections  of  required  JFS  activities  and 
available  resources,  then  revise  the  JFS  schedule 
accordingly.  The  revised  schedule  should 
identify  milestone  reviews  at  specific  phases  of 
the  JFS,  and  the  FSD  should  forward  the 
proposed  schedule  revisions  to  the  DDT&E 
within  60  days  after  issuance  of  the  JFS  directive 
for  approval.  The  progress  of  the  JFS  will  be 
based  on  the  approved  revised  schedule. 
Subsequently,  any  significant  changes  to  the 
schedule  should  again  be  coordinated  with  the 
DDT&E.  Figure  3-1  is  an  example  of  a  JFS 
WBS/schedule.  Milestone  reviews  are  suggested 
at  issues  solidification,  development  of  the  test 
concept,  and  upon  completion  of  the  APA  and 
final  report.  The  JFS  WBS  and  schedule  in 
Figure  3-1  must  be  tailored  to  the  specific 
requirement  of  each  JFS.  In  essence,  the  revised 
schedule  of  JFS  activities  and  milestones 
becomes  the  JFS  management  plan. 


3-2 


JFS  Handbook 
February  ly  1996 


JFS  ACTIVITY 


{Organize 

I  Facilities 
{  Staff 

Comm/ADP 

Contracts 

•  JFS  Plan/Schedule 
I  Formalize  Coord  Chain 
!  ; 

{Planning 

|  Determine  if  Necessary 
|  Develop  Concept 

Fully  Develop  Issues 
Scope 

Test  Scenario 
Measures/Data  Elements 
Data/Analysis  Methodology 
Program  Schedule 
Resource  Requirements 
|  Determine  if  Feasible 

|  APA 

Consider  Alternatives 
Expand  Concept  to  APA 

{  Final  and  Management  Report 

Prepare 

Coordinate 

Distribute 

|  Close  Down/Transition 


Tailor  JFS  Activities/Allocated  times 

•  Technical  Complexity 

•  Available  Staff 


Months 
5  6 


10 


::N:: 


M 


\ 


x 


\ 


■  Vi 


Figure  3-1.  Example  of  Joint  Feasibility  Study  WBS/Schedule 


D .  DETERMINE  OSD/JOINT 
STAFF/SERVICE 
COORDINATION,  SIGNATURE 
AUTHORITY,  AND  POCs 

One  of  the  first  requirements  for  the 
Service  deputies  will  be  to  determine  and 
establish  the  coordination  chain,  POCs, 
and  signature! coordination  authority  of 
their  Service  for  JFS  documents.  The 
Service  deputies  should  coordinate  with  their 
headquarters  JT&E  POCs  to  determine  the 
organizations  and  agencies  with  primary  interest 
in  the  JT&E.  They  then  should  coordinate  with 
these  organizations  and  agencies  to  determine 


their  POCs.  Each  of  the  POCs  should  be  asked 
to  determine  and  identify  who  within  their 
organization  or  agency  has  coordination  and 
signature  authority  for  JFS  documents.  The  FSD 
should  also  task  one  of  the  deputies  to  establish  a 
similar  OSD  and  Joint  Staff  coordination  chain. 
Once  formalized,  these  chains  should  be 
consolidated  into  a  published  formal  JFS 
coordination  chain.  The  chain  should  be  used 
exclusively  during  the  coordination  cycle  for  JFS 
documents.  Time  dictates  that  these  chains  be 
established  quickly.  Establishing  these  chains 
early  and  using  them  religiously  will  help  the  JFS 
to  achieve  its  goals  on  schedule. 
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A  complete  listing  of  POCs'  offices  on  the 
coordination  chain,  to  include  mailing  addresses, 
message  routing  addresses,  office  symbols,  and 
civilian  autovon  telephone  numbers,  is  an 
essential  part  of  the  process  and  must  be 
developed  as  soon  as  practical. 

The  FSD  should  realize  that  it  may  become 
necessary  to  revise  the  coordination  chains  as  the 
JFS  progresses.  The  FSD  should  welcome 
suggestions  and  ideas  from  all  involved 
organizations,  but  should  avoid  expanding  the 
formal  coordination  chain  outside  of  those 
directly  involved  or  affected  by  the  JT&E 
program  and  the  proposed  JT&E. 

A  major  problem  for  previous  JFSs  has 
been  that  as  new  personnel  and  organizations  are 
included  in  the  coordination  chain,  many  want  to 
restructure  the  program  to  include  special 
interests.  This  occurs  because  the  coordinating 
individuals  are  not  totally  familiar  with  the  JFS 
directive,  some  interpret  the  directive  differently, 
and  some  may  not  be  aware  of  previous 
discussions  and  negotiations.  Significant  time 
and  effort  have  been  expended  by  previous  JFSs 
responding  to  these  recommended  changes  and 
revisions.  Efforts  should  be  taken  to  avoid  these 
types  of  actions. 

E.  DETERMINE  IF  A  JT&E  IS 
NECESSARY 

The  next  step  would  be  to  determine 
whether  the  nominated  JT&E  is  necessary,  for  if 
it  is  determined  unnecessary,  there  is  no  reason 
to  carry  the  JFS  any  further.  JT&E  nominations 
can  vary  from  evaluations  of  concepts  or 
operational  use  of  new  or  improved  equipment, 
techniques,  and  procedures  to  proposed  solutions 


to  specific  problems.  Therefore,  the  following  is 
a  general  discussion  on  what  should  be  done  to 
answer  the  question,  Is  the  JT&E  necessary? 
This  question  should  be  resolved  by  evaluating 
the  nomination  in  terms  of  five  criteria,  and  the 
JFS  should  substantiate  and  document  that  each 
criterion  supports  conclusion  as  to  whether  a 
JT&E  is  necessary: 

1)  Do  the  nominated  concepts  or  issues  and 
objectives  meet  one  or  more  of  the  stated 
purposes  of  a  JT&E?  (See  Chapter  1, 
Section  B) 

2)  Are  the  issues  and  objectives  joint  and  do 
they  require  joint  efforts  and  solutions? 
While  the  nomination,  coordination,  and 
selection  process  should  preclude  non-joint 
issues  from  reaching  the  JFS  stage,  there  is 
the  possibility  that  an  issue  that  originally 
appeared  to  be  joint  may  subsequently 
prove  not  to  be.  Jointness  is  an  absolute 
requirement  and  is  not  waiverable. 

3)  Are  the  issues  being  addressed  elsewhere ? 
The  JFS  may  find  that  the  nominated  issues 
are  already  being  addressed  by  other 
agencies  or  that  these  agencies  could 
incorporate  the  issues  into  ongoing  or 
planned  activities. 

4)  Is  the  problem  of  sufficient  magnitude  or 
significance  that  it  warrants  expenditure  of 
the  estimated  resources? 

5)  How  likely  is  it  that  the  Services  will 
implement  or  use  the  JT&E  results? 
Identification  of  the  user  who  will  actually 
use  the  JT&E  products  is  essential. 

The  JFS  should  be  creative  in  determining 
how  each  of  the  five  criteria  will  be  derived  and 
used  to  resolve  the  question  of  necessity. 
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Does  the  nomination  meet  one  or 
more  of  the  stated  purposes  of  a  JT&E? 
The  JT&E  program  formally  brings  two  or  more 
Services  together  and  provides  specified  funding 
to  address  joint  matters  and  resolve  ;joint  issues 
that  cannot  be  effectively  addressed  or  resolved 
through  independent  Service  actions.  As 
discussed  in  Chapter  1,  the  purpose  of  the  JT&E 
program  is  to:  1)  assess  the  interoperability  of 
Service  systems  in  joint  operations  and  explore 
potential  solutions  to  identified  problems,  2) 
evaluate  and  provide  recommendations  for 
improvements  in  joint  technical  and  operational 
concepts,  3)  develop  and  validate  system 
development  and  testing  methodologies  having 
multi-Service  applications,  and  4)  evaluate 
technical  or  operational  performance  of 
interrelated/interacting  systems  under  realistic 
joint  operational  conditions.  This  aspect  of  joint 
testing  will  likely  become  more  important  as 
systems  and  sub-systems  are  upgraded  with  new 
technologies  or  as  new  technologies  are  fielded 
off  the  shelf. 

Is  the  nomination  joint?  Service 
downsizing  has  increased  the  probability  that 
U.S.  involvement  in  contingency  responses  will 
be  of  a  joint  nature,  and  support  of  these 
contingencies  requires  that  the  U.S.  develop 
more  flexible  joint  response  capabilities.  It  may 
be  necessary  for  theater  commanders  to  insert 
joint  responses  into  a  variety  of  situations  that 
could  range  from  disaster  relief  to  opposed 
military  operations.  In  fact,  joint  and  combined 
task  force  response  operations  now  dominate  our 
employments.  The  JFS  should,  therefore, 
evaluate  the  joint  aspects  of  the  nomination  in 
terms  of  potential  and  probable  joint  scenarios; 
for  example,  no  threat  situations  (disaster  relief), 
low  threat  (Grenada),  ambiguous  threat 


(Somalia),  or  opposed  operations  (Kuwait).  It  is 
probable  that  most  JT&E  nominations  will 
involve  military  operations  or  support  of  military 
operations  and  that  these  operations  will  be  joint. 

Can  the  nomination  be  addressed 
elsewhere?  The  JFS  must  determine  whether 
the  nominated  concepts  or  issues  have  been  or 
could  be  addressed  by  other  joint,  Service,  or 
OSD  agencies.  Most  joint  test  programs  could  be 
addressed  by  alternatives  such  as  a  Joint  Program 
Office  (JPO)  or  by  expanding  the  activities  of 
existing  OSD  or  Service  agencies  and 
organizations.  The  discriminator  for  a  JT&E  is 
that  it  involves  fielded  or  soon  to  be  fielded 
equipment  used  in  a  new,  enhanced,  or  broader 
joint  environment  or  concept  of  operation.  Once 
a  concept  has  been  proven  to  be  workable  by  a 
JT&E,  it  could  be  passed  to  an  acquisition  office 
for  applicable  development  and/or  acquisition. 

The  JFS  must  also  determine  if  the  issues 
of  the  nomination  have  been,  could  be,  or  are 
being  addressed  through  other  efforts  or  whether 
current  or  future  activities  of  these  organizations 
or  agencies  could  be  modified  to  address  the 
issues  of  the  nominated  JT&E. 

The  JFS  should  also  determine  whether  the 
nominated  issues  are  the  result  of  practices  or 
procedures  in  related  programs  that  could  be 
resolved  by  changes  in  those  programs.  For 
example,  a  nomination  involving  electromagnetic 
interference  problems  could  turn  out  to  be  the 
result  of  ineffective  frequency  allocations  during 
system  acquisition  or  the  result  of  ineffective 
frequency  assignment  and  control  during  field 
operations.  The  JFS  may  have  to  interface  with 
related  programs  to  identify  the  real  problem  and 
in  turn  address  the  most  realistic  solution.  The 


JFS  Handbook 
February  1, 1996 


3-5 


JFS  should  determine  not  only  whether  the 
nomination  can  be  addressed  elsewhere  but,  if 
so,  who  has  responsibility  for  resolution.  Some 
of  the  related  projects  may  involve  special  access 
programs,  in  which  case  the  FSD  must  rely  on 
the  DDT&E  staff  to  identify  and  assist  in 
obtaining  access. 

Does  the  proposed  JT&E  warrant 
expenditure  of  the  required  resources? 
JT&E  issues  are  usually  recommended  by 
organizations  that  have  seen  some  facet  of  the 
problem  but  may  not  have  a  global  understanding 
of  the  situation.  While  the  concepts,  issues,  and 
objectives  may  reflect  critical  unresolved 
problems,  the  proposed  solutions  may  reflect 
either  isolated  circumstances  or  parochial  interest 
or  biases.  Staffing  and  coordination  during  the 
nomination  process  provides  a  multi-Service 
perspective  of  the  nomination,  but  normally  not 
at  the  level  of  detail  that  is  required  to  fully 
evaluate  and  substantiate  the  nomination  in  terms 
of  the  magnitude  and  joint  significance  of  the 
situation  or  problem.  This  substantiation  should 
involve  determining  the  frequency  and 
distribution  of  the  deficiency.  The  JFS  should 
also  base  this  determination  on  the  probability 
and  number  of  occurrences  under  potential 
employment  situations.  Based  on  this  analysis, 
the  JFS  should  consider  whether  the  proposed 
concept  or  solutions  are  credible  and  warrant 
expenditure  of  the  estimated  resources  that  will 
be  required.  A  key  part  of  this  consideration  is 
developing  the  estimate  of  resources  required, 
which  is  discussed  in  detail  in  Chapter  4. 

Are  the  Services  and/or  Joint  Staff 
likely  to  implement  or  use  the  JT&E 
results?  This  criterion  is  an  assessment  of 
Service  support  expected  relative  to 


implementation  of  the  JT&E  proposed  solution. 
The  level  and  extent  of  Service  coordination 
during  the  nomination  process  is  a  starting  point 
for  this  determination.  The  proposed  solution 
may,  however,  involve  Service  or  Joint  Staff 
policy  or  doctrine  issues,  and  these  must  also  be 
considered.  For  the  most  part,  this  evaluation 
will  be  based  on  contacts  with  Service  and  Joint 
Staff  representatives,  inputs  from  the  Service 
deputies,  and  how  confident  the  FSD  is  in  the 
credibility  of  the  concept  or  solution  proposed  in 
the  nomination. 

During  this  evaluation,  the  JFS  should  also 
determine  what  the  users  require  as  a  results 
output.  Do  they  wish  results  only  or  do  they 
require  recommendations  for  revised  procedures, 
etc?  The  JFS  should  define  what  the  JT&E 
legacy  products  will  be  and  identify  who  will  be 
the  users  of  these  products.  Evidence  of  support 
to  use  the  JT&E  products  at  the  2/3/4  star  level  is 
necessary  and  is  not  difficult  to  obtain  if  the 
products  are  worthwhile. 

F .  FULLY  DEVELOP  THE  JT&E 

CONCEPTS,  ISSUES,  CRITERIA, 

AND  OBJECTIVES 

The  JFS  should  fully  develop  the 
nomination  concepts  and  issues  and  coordinate 
them  with  the  responsible  organizations  and 
agencies  that  are,  or  will  be,  involved  in  the 
JT&E.  This  is  the  most  critical  task  that  the  FSD 
will  undertake.  Failure  to  solidify  the 
concepts,  issues,  and  objectives  at  the 
very  start  of  the  JFS  will  impact  on  all 
subsequent  tasks  and  activities  the  JFS 
undertakes.  This  step  is  more  than 
fundamental,  it  is  essential.  The  JFS  will  simply 
bog  down  without  meaningful  progress  if  the 
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concepts  and  issues  are  subject  to  continual 
change  or  revision.  This  is  not  to  say  that 
concepts  and  issues  cannot  be  changed,  only  that 
such  changes  should  occur  only  when 
substantiated  as  necessary.  The  DDT&E,  with 
the  advice  of  the  TAB,  is  the  approval  authority 
for  all  such  changes. 

The  JT&E  issues  must  be  well  articulated  to 
convey  what  exactly  the  JT&E  plans  to  evaluate. 
The  JT&E  should  have  a  well  defined  concept 
that  it  plans  to  evaluate;  failure  to  articulate  a  well 
defined  concept  has  been  the  downfall  of  some 
JTFs. 

The  FSD  should  have  been  involved  in  the 
JT&E  PC  process  and  should  be  familiar  with  the 
nomination  concepts,  issues,  and  objectives  and 
the  coordination  actions  that  occurred  during  the 
nomination  process.  If  the  FSD  was  not 
involved  in  the  JT&E  PC,  he  must 
quickly  become  an  expert  on  the 
nomination  and  its  concepts,  issues, 
criteria,  and  objectives. 

Concepts 

Concepts  are  basically  perceived  ways  of 
doing  something  and  include  combinations  of 
systems,  equipment,  procedures,  and  personnel 
that  will  be  used  to  accomplish  the  intended  task. 
JT&Es  that  concentrate  on  concepts  involve 
equipment  and  systems  used  in  a  new  or  different 
environment,  procedure,  or  operation.  A  JT&E 
of  a  revised  concept  will  usually  include  a 
comparison  of  the  revised  and  the  old  concept 
and  the  quantification  of  observed  changes  in 
operational  capabilities.  Advances  in  technology 
frequently  provide  the  opportunity  to  develop 
new  concepts  with  significant  increases  in 


operational  capabilities,  reliability,  flexibility, 
and  mobility  while  reducing  resource 
requirements.  A  JT&E  of  a  new  concept  will 
also  include  an  evaluation  of  the  technical 
maturity  of  the  concept's  components,  the  risks 
associated  with  each,  and  a  projection  of  its 
capabilities  and  effectiveness. 

Issues 

Issues  are  questions  that  must  be  answered 
relative  to  concepts,  procedures,  system 
interfaces,  or  systems  capabilities  to  resolve 
multi-Service  problems  or  address  multi-Service 
needs.  They  address  the  tasks  and  areas  of  risk 
that  most  significantly  affect  mission 
accomplishment.  They  are  not  parameters, 
thresholds,  or  objectives.  Failure  to  resolve 
these  issues  renders  the  system  in  question 
unresponsive  to  the  needs  of  the  user.  Issues  are 
normally  written  as  specific  questions.  The 
issues  in  the  JFS  directive  reflect  the  problem  or 
concerns  as  seen  through  the  eyes  of  the 
nominator.  Joint  considerations  at  this  point  in 
the  JT&E  cycle  are  frequently  without  the  benefit 
of  a  detailed  review  or  study. 

Full  development  of  the  nominated  issues 
should  include  the  development  of  a  problem  or 
need  statement  and  an  issue  justification  paper 
that  substantiates  the  magnitude  and  scope  of  the 
problem.  The  FSD  will  find  that  a  clear  and 
concise  problem  statement  and  issue  justification 
paper  will  greatly  assist  in  the  coordination  of 
issues  and  objectives.  Formulation  of  the 
problem  statement  should  consider  all  aspects  of 
the  problem  to  include  doctrine,  policy, 
logistics,  and  operations.  Interfacing  with 
those  agencies  and  organizations  responsible  for 
these  functions  will  facilitate  the  development  of 
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a  realistic  definition  of  the  problem.  Once  the 
problem  has  been  identified,  the  solution  to  the 
problem  should  be  scoped  to  that  which  is 
meaningful  and  is  achievable  during  the  life  cycle 
of  a  JT&E.  The  issues  should  be  worded  in 
terms  that  are  simple  and  relevant. 
Decomposition  of  the  issues  into  subissues  or 
even  sub-subissues  may  be  necessary  so  that  all 
can  be  clearly  defined  in  terms  of  associated 
measures  and  tasks  that  must  be  accomplished. 
The  ideal  situation  would  be  one  issue  that 
defines  one  measure  that  defines  one  task.  This 
situation  seldom  occurs.  The  principle  of 
simplicity  and  direct  traceability  of  issues  to 
subissues,  to  measures,  to  tasks  should  be  used 
as  a  guide  in  the  full  development  of  issues. 

Criteria 

Criteria  are  statements  of  required  technical, 
effectiveness,  suitability,  and  supportability 
performance.  Criteria  are  frequently  expressed 
as  thresholds  that  provide  the  basis  for  collecting 
data  that  is  used  to  evaluate/answer  the  issues. 
Criteria  should  be  unambiguous  and  assessable, 
whether  stated  qualitatively  or  quantitatively. 
Criteria  are  the  values  deemed  necessary  by  the 
user  of  the  concept  or  system.  Criteria  are  used 
to: 

•  Support  measures  that  compare  the  mission 
performance  of  the  new  system  or 
procedures  to  that  being  replaced. 

•  Compare  new  concepts,  test  items,  or 
procedures  to  a  predetermined  standard. 

•  Compare  mission  performance  using  the 
new  concepts  and  test  items. 

•  Compare  procedures  to  operations  without 
them. 


Objectives 

Objectives  should  be  designed  to  define 
those  areas  in  the  JT&E  that  require  or  involve 
activities  to  gather  data  required  to  evaluate 
concepts  or  answer  issues  and  measures.  For 
example,  if  a  JT&E  issue  is  "What  is  the  best 
vehicle  for  a  specific  joint  force  element?"  the 
objective  will  be  "to  develop  and  execute  test 
activity  xx  that  provides  sufficient  data  to 
determine  what  is  the  best  vehicle  for  the  joint 
force  element."  When  the  JT&E  objectives  are 
clear,  the  test  methods  and  approaches  for 
achieving  the  objectives  will  be  understandable. 
Clear  objectives  define  the  nature  of  the  JT&E 
activities  and  influence  the  development  of  the 
test  measures. 

G.  COORDINATE  AND  SOLIDIFY 

THE  JT&E  CONCEPTS,  ISSUES, 

CRITERIA,  AND  OBJECTIVES 

The  JFS  must  coordinate  the  concepts, 
issues,  criteria,  and  objectives  with  the 
organizations  and  agencies  that  are  involved  in  or 
will  be  involved  in  the  JT&E  to  obtain  their 
concurrence.  This  coordination  may 
involve  considerable  negotiations, 
because  many  of  these  organizations  and 
agencies  will  attempt  to  restructure  the 
JT&E  and,  in  turn,  the  issues  during  the 
coordination  process.  A  technique  that  has 
been  used  to  minimize  the  effects  of  this  dilemma 
is  to  obtain  coordination  comments  from  each 
organization  or  agency,  insisting  that  the 
coordination  be  at  an  appropriate  level  to  speak 
for  the  organization,  incorporating  as  many 
comments  as  possible,  and  presenting  the  revised 
concepts  or  issues  and  objectives  to  all 
coordinators  at  a  conference  for  resolution  of  the 
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differences.  Coordination  is  easier  when 
conducted  under  conditions  where  differences  are 
aired  before  a  group  of  interested  and 
knowledgeable  representatives.  Of  importance 
here  is  that  the  conference  participants  be 
knowledgeable  of  the  proposed  JT&E  and  their 
organization's  original  input  and  be  able  to  speak 
for  their  respective  organizations.  The  FSD 
should  coordinate  in  advance  to  assure  the  right 
personnel  are  invited,  that  the  meeting  place  and 
date  are  convenient  and  acceptable,  and  that  the 
unit  representatives  understand  what  is  expected 
of  them. 

The  JFS  must  coordinate  the  review  of  the 
proposed  APA  and  associated  operational 
concepts  and  potential  changes  to  joint  and 
interservice  doctrine  with  the  Air  Land  Sea 
Agency  (ALSA)  and  the  Joint  Doctrine  Center 
(JDC).  Depending  on  the  magnitude  of  the 
impact  that  the  JT&E  could  have  on  doctrine, 
representation  from  ALSA  or  the  JDC  may  be 
desirable  or  required  at  the  General  Officer 
Steering  Committee  (GOSG)  or  other  appropriate 
advisory  group  or  panel.  The  FSD  and  JTD 
must  solicit  ALSA  and/or  JDC's  input  on  test 
results,  recommendations,  and  transition  of 
legacy  products  that  have  or  may  have  an  impact 
on  joint  doctrine,  tactics,  or  procedures. 

Once  the  concepts,  issues,  and  objectives 
have  been  fully  coordinated  through  the 
established  chains  of  command,  they  should  be 
approved  by  the  DDT&E  and  used  as  the 
foundation  for  all  subsequent  JFS  tasks, 
activities,  and  milestone  reviews.  OSD  level 
coordination  should  be  the  final  step  in  the 
coordination  process.  The  FSD  will  schedule  an 
in-process  review  (IPR)  to  review  the  study 


progress  and  obtain  the  DDT&E  approval  of  the 
fully  coordinated  concepts,  or  issues,  and 
objectives.  If  an  IPR  is  not  practical,  the  FSD 
may  submit  them  in  writing  directly  to  the 
DDT&E  for  approval. 

Remember,  these  approved  concepts, 
issues,  criteria,  and  objectives  become  the  basis 
of  all  future  JFS  activities,  reports,  and  reviews. 
This  is  not  to  say  that  changes  and  revisions 
should  not  be  considered.  However,  when 
changes  and  revisions  are  justified,  they  should 
be  coordinated  with  the  appropriate  OSD/Joint 
Staff/Service  organizations  and  agencies  and  be 
approved  by  the  DDT&E  so  that  focus  remains 
on  the  chartered  task.  If  significant  changes  or 
revisions  are  visualized,  and  they  will  impact  the 
JFS  schedule,  the  FSD  should  immediately 
apprise  the  DDT&E  of  the  situation  and  provide 
alternative  solutions. 

H .  SCOPE  THE  JOINT  TEST  & 

EVALUATION 

As  pointed  out  in  the  determination  of  "Is 
the  proposed  JT&E  necessary?"  the  test 
concept  and  APA  should  consider  all  facets  of  the 
nominated  concept  or  problem,  to  include  such 
things  as  deployment  phases  (in  garrison,  en 
route,  deployed),  levels  of  threat  (none,  low, 
ambiguous,  opposed),  etc.  These  considerations 
frequently  result  in  a  very  large  test  concept  that 
is  neither  practical  nor  achievable.  Down¬ 
scoping  the  JT&E  concept  or  proposed  solution 
may  be  required  to  assure  that  the  JT&E 
addresses  only  those  facets  of  the  problem  and 
objectives  that  are  germane  and  achievable  within 
the  prescribed  time  frame  and  available 
resources. 
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There  are  a  number  of  scoping  techniques 
that  have  been  successfully  used  in  previous 
JFSs.  Most  involve  dissecting  the  problem  to 
identify  and  isolate  those  critical  issues  that  relate 
directly  to  a  solution  to  the  problem.  The  JT&E 
test  concept  would  then  concentrate  on  those 
critical  issues.  Scoping  should  also  consider  test 
methods  that  make  maximum  use  of  available 
data  from  archives  or  other  test  sources.  Testing 
should  be  limited  to  those  situations  where  data  is 
essential  and  cannot  be  otherwise  obtained. 
Scoping  the  JT&E  concept  will  involve  two  other 
considerations:  assumptions  and  limitations/ 
constraints. 

Assumptions 

The  test  concept  cannot  be  designed  to 
include  all  possible  combinations  of  variables 
(targets,  weapons,  tactics,  command  and  control, 
threats,  environments,  weather,  light  conditions, 
etc).  The  JFS  must,  therefore,  develop 
assumptions  that  scope  the  JT&E  concept  to 
something  that  can  be  achieved  within  three 
years.  The  following  are  examples  of 
assumptions  from  several  previous  JT&Es  that 
can  be  considered  as  appropriate  for  this  scoping 
action: 

•  The  scenario  and  test  cell  matrix  bounds  the 
JT&E  to  critical  issues  and  achievable  test 
conditions. 

•  The  JT&E  concept  may  require  that 
surrogate  targets  and  weapons  be  used. 

•  The  JT&E  concept  may  involve  notional 
defenses  or  forces. 

•  Equipment,  techniques,  and  tactics  may  be 
limited  to  currently  fielded  systems  or 
represent  future  equipment/forces. 


•  The  JT&E  concept  may  address  only  certain 
weather  or  day /night  environments. 

•  The  JT&E  concept  does  not  attempt  to 
optimize  developing  techniques/ 
technologies. 

•  Live  ordnance  will  not  be  used. 

•  Kill  removal  is  not  a  concept  consideration 
for  this  JT&E. 

•  Test  participants  will  represent  fielded  units. 

•  Validated  modeling  and  simulation  inputs  or 
results  are  considered  realistic  and  will  first 
be  utilized  to  predict  actual  test  results  then 
in  lieu  of  or  to  extend  test  results. 

Limitations  and  Constraints 

Limitations  and  constraints  will  primarily 
be  related  to  real  world  test  environments 
(resources  are  limited,  scarce,  or  not  available; 
unacceptable  or  unavailable  training 
requirements,  range  scheduling;  operations; 
safety  and  environmental  concerns;  etc).  The 
following  are  examples  of  limitation  and 
constraint  considerations: 

•  A  full  factorial  evaluation  of  potential  test 
variables  is  not  possible.  Pairings  in  the  test 
cell  matrix  are  based  on  those  issues  that  are 
considered  to  be  the  most  important. 

•  Not  all  required  resources  are  available. 

•  Surrogate  resources  may  not  be  fully 
representative. 

•  Range  and  safety  limitations  preclude  full 
replication  of  combat  environments. 

•  Environmental  considerations  preclude  full 
use  of  a  technique  or  system. 

•  The  JT&E  concept  does  not  address  all 
systems  in  the  U.S.  or  threat  inventory. 

•  "Piggybacking"  on  training  activities  does 
not  totally  meet  all  test  requirements. 
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•  Funding  or  time  preclude  satisfying  all  test 
requirements. 

•  Available  technologies,  models,  or 
simulations  are  not  fully  mature  or  verified 
and  validated 

I .  DEVELOP  THE  JT&E  SCENARIO 

Most  nominations  are  based  on  operational 
situations,  and  the  resolution  of  the  identified 
problem  should  be  related  to  that  or  a  similar 
operational  and  environmental  situation.  The 
FSD  must  identify  a  test  scenario  for  the  test 
activities  that  can  be  directly  related  to  the 
environment  of  the  nominated  concepts  or  issues. 
The  scenario  must  consider,  as  a  minimum, 
operational  concepts  (if  an  issue  rather  than  the 
concept  is  being  studied),  force  structures  (size 
of  friendly  and  adversary  forces),  laydowns 
(realistic  force  component  locations),  rules  of 
engagement,  terrain,  tactics,  and  timing. 
Further,  the  scenario  should  be  based  on 
situations  that  conform  to  those  in  our  defense 
planning  guidance. 

There  are  a  number  of  sources  that  can  be 
used  to  select  or  establish  the  required  scenarios. 
The  Unified  Commands,  Joint  Publication  3-03 
( Joint  Interdiction  Operations)  and  the  Joint 
Military  Net  Assessment  are  sources  of  validated 
scenarios  that  can  be  adapted  to  fit  the  needs  of 
the  JFS.  The  use  of  an  existing  validated 
scenario,  or  a  subset  of  a  validated  scenario,  is 
recommended  whenever  possible.  Defense 
Intelligence  Agency  (DIA)  validation  of  a 
modified  scenario  is  required  and  may  take  some 
time.  If  none  can  be  found  that  satisfy  JT&E 
concept  requirements,  the  JFS  may  have  to 
modify  one  and  initiate  the  required  actions  to 
obtain  DIA  approval.  The  JFS  should 


avoid,  if  at  all  possible,  the  need  to 
develop  a  unique  scenario. 

Most  joint  test  scenarios  and  activities  will 
include  the  requirement  for  a  realistic  operational 
environment  that  involves  the  use  or  creation  of 
realistic  threats.  Here  again,  the  primary  source 
of  threat  information  is  DIA.  DIAR  55-3, 
Intelligence  Support  for  Defense  Acquisition 
Programs,  should  be  used  as  a  guide  to  obtain 
threat  information  support. 

J.  DEVELOP  THE  JT&E  CONCEPT 

Time  constraints  dictate  that  the  JFS  team 
quickly  concentrate  their  efforts  toward  a  singular 
goal:  to  develop  a  JT&E  concept  as  the  initial  step 
in  formalizing  the  APA.  The  JT&E  concept  is 
basically  a  structure  that  describes  the  who, 
what,  where,  when  and  why  of  the 
proposed  JT&E.  It  provides  the  links  between 
the  issues,  areas  of  risk,  measures,  test  methods, 
and  the  data  collection  and  analysis  process.  The 
JT&E  concept  is  used  as  the  vehicle  for 
determining  the  most  effective  way  to  execute  the 
proposed  joint  test.  When  the  JT&E  concept  is 
agreed  upon  by  the  TAB  and  the  DDT&E,  it  will 
be  expanded  upon  and  formalized  to  become  the 
APA.  A  structured  approach  should  be  used  to 
develop  a  concept  that  will  accomplish  the  JT&E 
objectives  and,  in  turn,  resolve  the  JT&E  issues. 
This  structured  approach  begins  with  the 
definition  and  development  of  the  evaluation 
methodology.  This  methodology  will  establish  a 
high  level  definition  and  agreement  among  JFS 
participants  and  supporting  agencies  of  the 
general  and  realistic  parameters  of  the  supporting 
evaluation.  This  methodology  will  be  a  useful 
tool  to  bound  the  definition  of  test  measures  and 
subsequent  waterfall  of  test  details.  This  process 
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will  involve  a  number  of  closely  related  activities 
that  must  be  considered  and  developed 
simultaneously.  These  are: 

•  A  rigorous  translation  of  the  issues  to  test 
measures  and,  in  turn,  to  test  data  elements 
(dendritic  structure). 

•  An  approach  for  resolving  the  JT&E  issues. 

•  The  development  of  a  test  method  matrix. 

•  The  development  of  a  test  cell  matrix. 

Figure  3-2  is  an  example  of  a  JT&E 
concept  that  shows  the  overall  relationship 
between  the  issues,  test  methods,  and  products. 
Figure  3-3  is  an  overview  of  the  steps 
commonly  used  in  the  concept  development 
process.  The  JFS  team  will  use  the  JT&E 
concept  to  define  the  analysis  techniques  and 
methodologies  that  will  be  used  for  consolidating 
the  test  results  to  resolve  the  JT&E  issues. 

Develop  the  Dendritic  Structure 

The  dendritic  process  is  nothing  more  than 
a  structured  process  for  identifying  all  of  the 
elements  or  actions  necessary  for  addressing  or 
resolving  the  concept  or  system  issues.  The 
process  also  provides  the  underlying  structure  for 
the  program  level  analysis  approach  and 
methodology.  A  key  component  to  the  process  is 
linkage :  all  data  collected  using  the 
various  test  methods  is  required  and  has 
a  place  in  satisfying  the  issues.  This 
linkage  assures  that  only  required  data  is  planned 
for  and  that  resources  will  not  be  wasted  by 
planning  unnecessary  activities  or  collecting 


unnecessary  data.  Verification  of  this  process  is 
sometimes  known  as  a  data  trace.  An  incomplete 
dendritic  structure  may  not  yield  enough 
information  to  adequately  address  the  issues. 
Worse  yet,  an  incomplete  structure  may  result  in 
false  conclusions  drawn  from  incomplete  sets  of 
information. 

To  develop  the  dendrite,  the  JFS  team  must 
"decompose"  the  issues  to  their  lowest  level  by 
asking  "What  must  I  know  to  address  or  resolve 
each  issue?"  The  answers  have  attributes  that  can 
be  used  to  develop  the  next  level  of  questions. 
When  the  attributes  reach  a  level  that  can  be 
quantified,  this  next  level  is  called  measures.  The 
criteria  for  quantifying  these  measures  are 
expressed  in  terms  of  what  the  system  or  concept 
is  required  to  do.  Three  levels  of  measures  are 
commonly  used  to  relate  system  or  concept 
attributes  to  the  required  task  to  be  performed:  the 
overall  mission,  systems  capabilities  in  terms  of 
task  accomplishments,  or  systems  capabilities  in 
terms  of  characteristics  or  attributes. 

For  each  measure,  the  next  level  is  the  data 
elements  required  to  calculate  the  measure.  When 
this  data  element  level  is  reached,  What  I  must 
know?  is  a  measurable  quantity.  This 
measurement  can  be  obtained  either  from  existing 
data  resources  or,  if  truly  unique  and  unavailable, 
through  some  degree  of  actual  testing.  This 
process  of  breaking  the  issues  down  to  the  data 
element  level  represents  the  development  of  a 
dendritic  structure.  Figure  3-4  is  an  example 
of  a  simple  dendritic  structure. 
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Figure  3-2.  Example  of  a  JT&E  Concept 


Since  the  data  elements  for  the  dendritic 
structure  constitute  the  entire  data  requirements 
for  the  JT&E,  the  dendritic  process  requires  a 
thorough,  in-depth  knowledge  of  all  concepts  or 
issues  in  order  to  identify  all  the  information 
necessary.  For  problem  solving  issues,  this 
equates  to  an  organized  sequential  approach  for 
gathering  needed  data,  developing  hardware  or 
software  processes,  implementing  problem 
solving  techniques,  or  combinations  of  these 
approaches. 


Inherent  in  the  process  of  developing  the 
dendrite  is  the  identification  of  the  analysis 
methods  required  to  convert  data  elements  into 
measures  and  then  to  information  required  to 
answer  questions  at  the  issue  level.  The  JFS 
must  document  the  required  analysis  methods  as 
the  dendritic  structure  is  developed  because  they 
form  the  basis  for  the  program-level  analysis. 
Further,  the  complexity  of  the  program-level 
analysis  will  likely  parallel  the  complexity  of  the 
dendritic  structure. 
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Figure  3-3.  Examples  of  Steps  in  the  JT&E  Concept  Development 


Develop  the  JT&E  Measures 

Annex  E  provides  a  more  detailed 
explanation  of  the  most  commonly  used  process 
to  analyze  and  fully  develop  the  test  issues.  In 
this  process,  the  JFS  will  develop  criteria  and 
measures  for  each  of  the  issues  or  subissues  as  a 
way  of  relating  the  results  of  testing  to  resolve 
the  issues.  There  are  three  commonly  used  types 
of  measures:  mission  level  measures  (MLMs) 
relate  the  effects  of  a  concept  or  system  to  the 
mission  that  the  concept  or  system  supports; 
measures  of  effectiveness  (MOEs)  are  used  to 
measure  a  system's  effectiveness  in  the 
accomplishment  of  a  task;  and  measures  of 
performance  (MOPs)  are  used  as  a  quantitative  or 
qualitative  measure  of  system  or  system 
component,  capabilities  or  characteristics.  All 


measures  must  be  derived  from  quantifiable  test 
data  that  characterizes  the  extent  to  which  some 
test  objective  has  been  achieved.  A  description 
of  each  measure  should  be  developed  that  defines 
specifically  how  it  will  be  calculated. 

Mission  Level  Measures  measure 
concept  or  system  capabilities  in  terms  of  the 
effects  of  these  capabilities  on  the  overall  mission 
of  which  the  concept  or  system  is  a  part.  Most 
will  be  measures  of  mission  or  system  attributes 
that  must  be  defined  in  terms  of  measurable 
qualities.  An  example  of  an  attribute  for  a 
command  and  control  (C2)  system  would  be 
reliability,  and  an  example  of  a  measure  for 
reliability  could  be  operational  failure  mean  time, 
operational  mission  mean  time,  unscheduled 
maintenance  mean  time,  etc. 
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Is  Telephone 
System  Working? 


I  •  Do  Calls  Come 

•  Line  Quality  •  %  of  Words  Heard 

|  In? 

•  %  of  Words  Heard 

| 

Correctly 

|  •  Can  I  Place 

j  Outgoing  Calls? 

! 

i  E 

•  Number  of  Words  Spoken 

•  Number  of  Words  Heard 

•  Number  of  Incorrect  Words  Heard 

•  Number  of  Words  Repeated 


Figure  3-4.  Example  of  a  Dendritic  Structure 


Measures  of  Effectiveness  measure 
concept  or  system  capabilities  in  terms  of  task 
accomplishment  or  system  attributes.  Most 
JT&E  nominations  involve  concepts  or  systems 
that  can  be  related  directly  to  operational 
capabilities  in  terms  of  engagement  or  battle 
outcome.  For  those  systems  that  cannot  be 
directly  related  to  operational  capabilities,  the 
developed  measures  should  be  expressed  in 
terms  of  concept  or  system  attributes.  For 
example,  C2  system  measures  can  be  expressed 
in  terms  of  capacity,  consistency,  timeliness, 
accessibility,  completeness,  accuracy, 
transportability,  and  security.  The  measures 
should  then  be  developed  to  levels  of  specificity 
so  that  the  C2  system  can  be  evaluated  both  in 
terms  of  effectiveness  and  suitability.  MOE 
evaluation  criteria  should  be  quantitative  if  at  all 
possible. 


Measures  of  Performance  are 

quantitative  or  qualitative  measures  of  a  system's 
capabilities  or  characteristics.  They  are  used 
when  it  is  difficult  to  directly  test  an  MOE  or  to 
establish  quantitative  criteria.  In  these  cases, 
MOPs  are  developed  that  can  be  aggregated  at  the 
MOE  or  issue  level.  MOPs  should  be  quantitative 
whenever  possible.  Quantitative  MOPs  are 
measures  of  system  performance  that  can  be 
related  to  some  numerical  scale;  for  example,  a 
communication  system  throughput.  Qualitative 
MOPs  are  categorical  measures  of  system 
performance  that  refer  to  the  presence  or  absence 
of  system  characteristics.  Usually  subjective 
measurement  techniques  are  used  to  address 
qualitative  MOPs.  Qualitative  MOPs  are  based 
on  system  characteristics  that  are  defined  by  the 
needs  of  the  user  or  meet  user-defined  system 
performance  requirements. 
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Derived  Measures,  in  some  cases,  may 
be  required  to  resolve  an  issue.  Derived 
measures  result  from  analysis  or  manipulation 
techniques  (such  as  simulations,  aggregations  of 
many  scores,  extrapolations,  etc).  They  may  be 
the  only  or  most  effective  way  to  fully  address  a 
JT&E  issue.  For  example,  JT&E  tests  that  do 
not  employ  live  ordnance  could  include  the  use  of 
the  Joint  Munitions  Effectiveness  Manuals 
(JMEM)  to  convert  weapons  miss  distances  to 
target  damage  estimates  and,  in  turn,  to  target 
survivability. 

Define  the  Required  Data  Elements 

A  thorough  development  of  the  JT&E  measures 
will  allow  determination  of  the  specific  data 
elements  that  will  be  required  to  answer  the 
measures.  Normally,  these  data  elements  will 
remain  constant  between  test  activities,  but  the 
methods  for  obtaining  the  data  may  vary.  The 
JT&E  concept  must  consider  the  required 
capabilities  to  acquire  this  data  in  terms  of 
source,  type,  format,  sample  rates,  and  collection 
methods.  It  must  also  consider  the  availability  of 
data  extraction  points,  instrumentation,  test 
controls,  documentation  of  uncontrolled  test 
variables,  and  the  interface  that  will  be  required 
between  instrumentation,  the  test  facility,  and  the 
JT&E  test  team. 

As  mentioned  previously,  the  individual 
data  elements  are  derived  through  a  dendritic 
process  that  divides  issues  into  subissues  and 
subissues  into  progressively  finer  subdivisions 
until  directly  measurable  data  elements  emerge. 
Ordinarily,  this  results  in  a  large,  complex  root¬ 
like  structure.  It  is  usually  most  convenient  to 
conduct  this  iterative  process  in  the  form  of  a 
blackboard  exercise  because  of  the  numerous 


changes  that  are  expected  during  the  initial 
breakdown.  Conducting  this  exercise  is  usually 
one  of  the  more  demanding  tasks  of  the  JT&E 
process.  When  the  final  derivation  of  data 
elements  is  complete,  the  FSD  selects  data 
elements  necessary  for  testing. 

Often,  one  aspect  of  an  issue  has  several 
possible  data  sources,  with  not  all  of  them  being 
needed.  However,  some  redundancy  is 
desirable,  especially  when  a  data  element  can 
only  be  gathered  by  sophisticated  instrumentation 
that  may  be  subject  to  failure.  In  most  cases,  this 
instrumentation  would  be  backed  up  with  a 
simpler,  manual  measure.  Additionally,  there 
may  be  duplication  of  data  elements  obtained 
while  exploring  very  different  aspects  of  the 
JT&E  concepts  and  issues.  This  duplication  may 
be  desirable  in  case  the  primary  source  should 
fail  or  the  testing  be  canceled  or  changed.  As 
stated,  the  selection  of  data  elements  to  be  tested 
is  an  especially  demanding  task.  It  also  involves 
the  risk  of  selecting  inferior  data  elements. 
Therefore,  it  is  advisable  to  seek  the  aid  of 
specialists  and  subject  matter  experts  (SME) 
during  this  process.  It  is  necessary  that  the  data 
elements  be  defined  in  sufficient  detail  to  support 
the  identification  and  quantification  of  JT&E 
resource  requirements. 

The  above  process  also  attempts  to  resolve 
the  three  points  that  are  critically  relevant  to  a 
JT&E:  adequacy,  quality,  and  credibility. 
Adequacy  refers  to  completeness  in  addressing 
the  JT&E  issues  or  concepts.  If  a  JT&E  failed  to 
address  the  issues,  it  was  a  failure  in  that  an 
amount  of  uncertainty  associated  with  the 
concepts,  procedures,  system  interfaces,  or 
system  capabilities  was  not  identified  and 
reduced  or  corrected.  Quality  is  associated  with 
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the  relevance  of  the  issues  addressed.  Right 
answers  to  wrong  questions  must  be  avoided.  If 
improper  or  inadequate  data  are  chosen,  the 
JT&E  will  not  provide  the  necessary  information 
to  decision  makers.  Tests  will  have  to  be  redone, 
causing  both  schedule  and  cost  impacts.  This 
represents  significant  risk  to  the  program. 
JT&Es  must  provide  information  that  is  clear  and 
accurate  beyond  a  reasonable  doubt.  Credibility 
refers  to  the  degree  to  which  issues,  concepts, 
procedures,  or  system  capabilities  are  examined 
and  assessed.  Proper  test  planning  and  conduct, 
along  with  aggressive  follow-on  evaluation,  are 
critical  to  supporting  credibility. 

Define  the  Evaluation  Procedures 

Evaluation  procedures  are  the  methods  used 
to  reduce  the  data  to  information  that  decision 
makers  can  use.  Concepts  and  issues  that  must 
be  addressed  by  a  JT&E  are  often  distanced  from 
the  actual  data  collected.  In  effect,  specific  and 
valid  evaluation  procedures  tie  the  tiers  of  the 
dendritic  hierarchy  together,  ultimately  to  form  a 
critical  link  between  the  top  tier  (concepts  and 
issues)  and  the  bottom  tier  (data  elements). 
Without  this  link,  the  data  collected  is  useless. 
Statistical  processes,  numerical  analyses,  and 
subjective  assessments  are  all  legitimate 
evaluation  methods;  however,  no  method  is 
universally  applicable  or  reasonable.  For 
example,  a  test  activity  that  collects  a  large 
number  of  data  points  typically  lends  itself  to 
statistical  processes,  yet  a  test  activity  that  yields 
a  single  data  point  is  virtually  impossible  to 
evaluate  with  statistics,  but  is  nonetheless 
significant.  In  the  single  data  point  case,  a 
subjective  assessment  may  be  the  only  viable 
evaluation  procedure. 


The  reality  is  that  the  complexity  of  the 
hierarchy  from  concepts  or  issues  to  data 
elements  drives  a  corresponding  complexity  in 
evaluation  procedures.  The  data  derived  at  the 
lowest  level  must  be  evaluated  to  arrive  at  a  result 
or  factor  that,  when  combined  with  other  test 
results  or  factors,  must  likewise  be  analyzed  and 
evaluated.  This  process  is  repeated  at  each  step 
upward  through  the  hierarchy  to  arrive  at 
evaluated  information  that  the  decision  makers 
can  act  upon  and  that  is  applicable  to  the  issue  or 
concept  as  a  whole. 

Develop  an  Approach  for  Addressing  the 
Concepts  or  Issues 

The  JFS  should  develop  an  approach  for 
addressing  each  of  the  nominated  concepts  or 
issues.  These  approaches  should  address  the 
requirements  for  test  activities  in  the  three  phases 
of  all  tests:  pretest,  test,  and  posttest. 
Figure  3-5  is  an  example  of  an  approach  for 
addressing  a  test  issue.  The  pretest  phase 
identifies  and  defines  those  activities  required  to 
conduct  the  test  activity,  establishes  the  test 
parameters,  and  identifies  instrumentation  and 
support  requirements.  The  test  phase  identifies 
and  defines  those  activities  associated  with  test 
execution  and  the  collection  of  data  required  to 
support  the  test  measures.  The  posttest  phase 
identifies  and  defines  those  activities  associated 
with  the  reconstruction  and  validation  of  the  test 
data,  the  analysis  of  the  data  collected,  and  how 
the  analysis  will  be  used  to  address  the  concepts 
or  issues  being  studied. 
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Figure  3-5.  Example  of  an  Approach  for  Addressing  an  Issue 


Define  Test  Methods 

Each  of  the  required  activities  in  the  JT&E 
concept  should  be  carefully  scrutinized  to 
determine  how  the  desired  results  can  best  be 
obtained.  All  available  test  methods  (studies, 
analyses,  simulations,  models,  laboratory  tests, 
mini-tests,  and  field  tests)  should  be  considered. 
Selecting  the  best  test  method  for  a  test  activity 
will  be  based  on  considerations  such  as  the 
availability  and  credibility  of  the  test  method, 
expected  acceptance  of  test  results  by  the 
Services,  fidelity  of  acquired  data,  availability  of 
test  site  location,  cost,  and  support. 

When  the  appropriate  test  method  has  been 
identified  and  substantiated  for  each  test  activity, 
the  methods  should  be  incorporated  into  a  test 
method  matrix  that  identifies  each  test  method 


and  rates  its  cost  and  risk  factors  relative  to  a 
specified  test  activity.  The  cost  and  risk  factors 
will  be  based  of  the  JFS  evaluation  of  each 
method  and  should  be  portrayed  in  a  manner  that 
is  consistent  with  the  fidelity  of  the  evaluation.  In 
some  cases,  the  matrix  will  have  to  be  expressed 
in  terms  of  low,  medium,  and  high  cost/risk. 
More  comprehensive  evaluations  may  provide  the 
capability  to  express  these  evaluations  in  terms  of 
numerical  ratings;  for  example,  a  rating  scale  of 
1-10.  Figure  3-6  is  an  example  of  a  test 
method  matrix.  When  the  appropriate  test 
methods  have  been  identified,  they  will  be 
incorporated  into  the  test  concept.  The  matrix 
will  also  be  used  in  the  consideration  of 
programmatic  and  technical  alternatives.  A  more 
detailed  discussion  of  test  methods  can  be  found 
in  Annex  F. 
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TEST  METHODS 
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Figure  3-6.  Example  of  Test  Method  Matrix 


Define  Test  Cell  Composition 

A  full  factorial  evaluation  during  a  JT&E  is 
seldom  feasible  because  of  time  and  cost 
constraints.  A  test  cell  matrix  should  be 
developed  to  assure  that  the  most  significant 
combinations  of  controlled  test  variables  are 
addressed  in  the  JT&E  concept.  This  matrix  links 
the  JT&E  measures  to  specific  test  activities  and 
test  conditions  that  will  produce  the  data  required 
to  resolve  the  program  issues.  The  test  cell 
matrix  represents  a  balance  between  test  items, 
test  environments,  test  conditions,  and  controlled 
test  variables.  Potential  uncontrolled  variables 
should  be  identified  so  that  provisions  for 
documenting  their  occurrence  can  be  incorporated 
into  the  JT&E  concept. 


Each  test  cell  in  the  test  cell  matrix 
constitutes  a  stand-alone  experiment  of  test  items 
and  procedures  under  specific  conditions. 
Figure  3-7  is  an  example  of  a  test  cell  matrix. 

The  above  components  are  the  basis  for  the 
development  of  the  JT&E  concept.  The  JT&E 
concept  should  be  reviewed  by  the  TAB  at  the 
second  JFS  milestone  to  verify  its  technical 
credibility  and  to  document  its  acceptance  by  the 
participating  Services.  This  milestone  review  also 
allows  for  changes  or  revisions  prior  to  the 
formalization  of  the  APA. 
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Figure  3-7.  Example  of  a  Test  Cell  Matrix 


K.  DEVELOP  A  JT&E  DATA 
MANAGEMENT  AND  ANALYSIS 
METHODOLOGY 

Planning  for  data  management  and  analysis 
is  not  a  distinct  activity,  but  rather  flows  directly 
from  the  requirements  interrelationships  and 
between  the  test  design,  issue  resolution 
approach,  test  methods,  and  measures. 
Depending  on  the  type  of  test  activities  and 
events  and  the  amount  and  type  of  data  to  be 
collected,  data  management  may  be  simple  and 
straightforward,  or  complex  and  resource 
intensive. 


The  JFS  is  responsible  for  laying  out  the 
approach  and  framework  for  these  activities  in 
sufficient  detail  so  that  it  is  possible  to  judge  the 
impact  of  data  management  and  analysis  activities 
on  the  overall  cost  and  scheduling  of  the  program 
as  part  of  the  chartering  decision.  A  good  place 
to  start  is  to  look  at  how  data  management  and 
analysis  were  carried  out  in  other  joint  tests 
involving  similar  kinds  of  issues  and  data 
collection  requirements.  DESA  is  the  repository 
for  hardware  and  software  used  by  previous 
JT&Es  and  should  be  able  to  provide  the  JFS 
with  information  relative  to  current  capabilities 
and  possible  solutions  to  unique  challenges  in  the 
data  management  and  analysis  area. 
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Data  Collection 

Data  collection  encompasses  what  will  be 
collected,  how  frequently,  and  by  what 
means.  They  all  affect  the  feasibility  and  cost 
associated  with  the  program  test  design.  The  JFS 
will  have  to  determine  which  data  collection 
efforts  can  be  provided  through  existing  range  or 
facility  instrumentation  (e.g.,  TSPI,  GPS,  etc.) 
as  opposed  to  those  requiring  specialized 
equipment  that  must  be  brought  to  the  test 
location.  Manual  data  collection  methods  (i.e., 
forms,  logs,  questionnaires,  etc.)  will  have  to  be 
considered  because  most  tests  require  at  least 
some  collection  of  this  type  of  data.  In  some 
tests,  manual  collection  forms  may  even  be  the 
primary  source  of  data.  Additionally,  many 
different  kinds  of  "record  keeping"  equipment 
(tape  recorders,  infrared,  video  and  still  cameras, 
meteorology  stations,  etc.)  may  be  required  as 
either  primary  or  supplemental  data  sources.  All 
of  the  above  point  to  the  importance  of 
considering  the  entire  data  collection  process  in 
calculating  the  cost,  equipment,  and  personnel 
needs  of  a  proposed  joint  test.  Some  of  the 
required  data  collection  devices  may  be  "long- 
lead"  items  that  will  necessitate  immediate  action 
by  a  JTF  upon  charter. 

Data  Management 

Data  management  encompasses  all  of  the 
control  processes  and  hardware  and  software 
involved  in  collecting,  storing,  and  retrieving  test 
data.  Although  it  is  conceivable  that  a  test  may 
involve  only  a  manual  data  management  system, 
it  is  much  more  likely  that  the  data  management 
system  will  be  computerized.  Thus,  computer 
hardware  and  software  requirements  must  be 
addressed  early  on.  Hardware  choices  will  be 


driven  by  such  factors  as  data  storage 
requirements,  security,  turnaround  time,  number 
of  users,  and  networking  requirements. 
Software  needs  can  generally  be  met  with 
commercial  off-the-shelf  relational  database 
systems,  although  some  development  of 
specialized  preprocessing  software  may  be 
required.  Increasingly,  multimedia  databases  that 
can  store  video,  audio,  and  image  files  are 
becoming  available  at  reasonable  cost.  This 
equipment  makes  the  integration  of  different 
types  of  test  data  possible  and  can  greatly  add  to 
the  tools  available  for  analysis  and  reporting. 

Many  factors  such  as  processing  time  and 
user  expertise,  determine  how  much  development 
will  go  into  automating  and  customizing  such 
functions  as  data  entry,  error  checking,  database 
querying,  and  report  generation.  DESA  and  the 
Service  test  resource  managers  can  provide 
valuable  information  and  guidance  to  the  JFS 
based  on  experience  with  many  kinds  of  tests  and 
a  wide  range  of  hardware  and  software  tools. 

Data  Analysis 

Data  analysis  software  needs,  in  many 
cases,  can  be  met  by  the  database  itself  or  with 
the  analysis  functions  available  in  easy-to-use 
spreadsheet  software.  If  statistical  analysis  is 
required,  it  will  usually  be  advisable  to  have  a 
statistical  software  package  available.  Graphics 
capabilities,  tables,  and  flowcharting 
requirements  must  also  be  addressed  and  a 
determination  made  as  to  whether  specialized 
software  must  be  purchased,  or  whether  database 
or  analysis  software  can  be  adapted  to  satisfy 
these  needs.  In  addition,  consideration  should  be 
given  to  analysis  requirements  for  the  various 
other  media  that  may  be  required.  For  example,  if 
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audio  and  video  data  are  expected  to  be  extensive 
or  serve  as  the  primary  data  source,  a  tool  such 
as  a  digital  audio  processing  system  should  be 
considered. 

Figure  3-8  is  an  example  of  a  flow 
process  that  has  been  used  by  previous  JT&Es  to 
develop  data  management  and  analysis 
methodologies.  Whatever  the  decision,  the 
selected  analysis  methodology  should  address  all 
aspects  of  issue  resolution  to  include 
considerations  of  how  the  data  will  be  used  to 
provide  results,  conclusions,  and 
recommendations.  The  methodology  must  also 
consider  the  possible  requirements  to  access 
various  formatted  data  sources;  respond  to 
changes  in  instrumentation  sources,  formats,  and 


contents;  track  entire  data  sets  and 
documentation;  and  extract  patterns  of  system 
behavior  from  large  data  sets. 

Analysis  Planning 

Planning  for  the  analysis  of  the  collected 
data  is  an  iterative  process  that  involves  both 
formal  and  informal  analysis  of  the  employed 
concept  and  system  capabilities  (threat, 
environment,  tactics,  interoperability,  etc.)  in 
terms  of  the  mission  that  it  supports.  Techniques 
that  have  proven  effective  in  the  development  of 
analysis  plans  include  process  analysis, 
engineering  analysis,  matrix  analysis,  systems 
analysis,  and  structured  analysis. 


Figure  3-8.  Example  of  a  Data  Management  and  Analysis  Flow  Process 
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Process  analysis  consists  of  thinking 
through  how  the  concept  or  system  will  be  used 
in  the  projected  environments  (threats,  missions, 
and  scenarios)  in  order  to  understand  and 
evaluate  the  actions,  events,  situations,  and 
results  that  are  expected  to  occur.  It  involves 
decomposing  the  objectives  to  be  achieved  into 
the  processes  or  actions  that  must  be 
accomplished.  These  processes  are  then 
translated  into  system  characteristics  or  attributes 
that  can  be  measured  by  the  test  activities.  This 
technique  is  most  often  used  when  the  JT&E 
involves  methodologies  and  concepts.  Process 
analysis  measures  are  frequently  based  on 
concept  or  system  attributes. 

Engineering  analysis  consists  of 
evaluating  the  mechanical  or  functional 
operations  of  a  system.  This  technique  involves 
a  systematic  exploration  of  the  system  hardware 
and  software  components,  purpose,  performance 
bounds,  manpower  and  personnel 
considerations,  known  problem  areas,  and 
impacts  on  other  system  components. 
Exploration  of  the  way  the  system  operates  as 
compared  to  intended  performance  is  used  to 
identify  issues,  measures,  specific  data  elements, 
test  activities,  and  required  instrumentation. 
Measures  to  support  this  type  of  analysis  are 
usually  based  on  system  or  component 
specifications,  or  expected/projected  per¬ 
formance. 

Systems  analysis  consists  of 
determining  the  interdependencies  and 
interactions  between  system  components.  This 
technique  is  used  to  define  the  relationships 
between  system  characteristics  to  optimize  the 
design  within  an  overall  system  concept. 


Structured  analysis  consists  of 
decomposing  the  issues  to  the  point  that  test 
activities  required  to  obtain  the  data  elements  can 
be  identified.  This  is  the  most  commonly  used 
of  the  analysis  techniques.  Annex  E  provides  a 
detailed  explanation  of  the  structured  analysis 
approach  and  process. 

Matrix  analysis  is  useful  to  diagram  the 
interactions  or  interdependencies  of  the  measures 
and  serves  as  an  organizing  concept  to 
characterize  problems.  This  technique  can  be 
useful  for  tracing  a  system's  operational 
requirements  through  issues,  criteria,  sources  of 
individual  data  requirements,  and  test  activities. 
Measures  to  support  this  type  of  analysis 
generally  consist  of  parametric  evaluations  of 
multiple  system  or  component  functions. 

L.  DEVELOP  THE  JT&E  SCHEDULE 

The  joint  test  concept  will  identify  the 
required  test  activities  and  events  that  must  be 
accomplished  within  the  three  years  allotted  for  a 
JT&E.  The  JFS  must  incorporate  these  test 
activities  into  a  program  schedule  that  provides 
for  completion  of  the  test  activities  in  sufficient 
time  to  accommodate  analysis  and  reporting 
requirements.  The  test  method  matrix  identified 
the  best  test  methods  for  accomplishing  each  of 
the  test  activities.  The  JFS  must  determine  where 
and  when  these  test  activities  can  best  be 
accomplished.  Interfacing  with  OSD,  Joint 
Staff,  and  the  Services  is  required  to  identify 
training  exercises,  command  post  exercises,  and 
other  test  events  that  might  be  used  to  satisfy  the 
joint  test  requirements.  Figure  3-9  is  an 
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Figure  3-9.  Example  of  a  Test  Opportunity  Schedule 


example  of  a  test  opportunity  matrix  that 
identifies  scheduled  Joint  Staff  and  Service 
activities  that  might  be  used  to  satisfy  JT&E 
requirements. 

Interfacing  with  the  controlling  agencies  of 
the  scheduled  events  will  determine  if  and  when 
participation  is  possible.  If  there  is  no  possibility 
of  using  already  scheduled  events,  the  JFS  will 
have  to  consider  conducting  dedicated  test 
activities.  When  agreement  has  been  reached  that 
the  JT&E  can  participate  in  the  required  events, 
the  JFS  should  consolidate  the  potential  events 
and  the  planned  joint  test  activities  into  a  JT&E 
test  schedule.  Figure  3-10  is  an  example  of  a 
generic  test  schedule  that  provides  an  overview 
of  the  potential  for  accomplishing  the  planned  test 
activities. 


M.  DETERMINE  IF  THE  PROPOSED 
JT&E  IS  FEASIBLE 

Determining  whether  the  JT&E  is  feasible 
is  more  complex  than  determining  whether  the 
JT&E  is  necessary.  Five  questions  must  be 
answered,  each  bearing  on  the  determination  of 
feasibility: 

•  Are  the  resources  (personnel,  ranges,  test 
items,  etc.)  available? 

•  Can  the  JT&E  be  completed  in  three  years? 

•  Are  the  benefits  gained  worth  the  cost? 

•  Can  a  solution  be  found  and  do  the 
technologies  exist  to  support  the  solution? 

•  Is  joint  testing  the  most  effective  way  to 
resolve  the  concepts  or  issues? 
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Figure  3-10.  Example  of  a  JT&E  Program  Schedule 


I  Mini -Test 


The  FSD  must  plan  and  manage  the  JFS 
activities  so  as  to  obtain  the  required  information 
and  data  to  answer  these  questions.  The  source 
and  accuracy  of  the  information  and  data  will 
determine  the  credibility  of  the  answers  and 
recommendations  that  will  be  developed.  The 
FSD  must,  therefore,  decide  early  in  the  JFS 
what  and  how  much  information  is  required  to 
assess  the  questions  that  support  the 
determination  of  is  the  JT&E  feasible? 

Are  the  resources  available?  The 
procedures  to  quantify  the  required  resources  to 
accomplish  the  proposed  JT&Es  mission  are 
discussed  in  Chapter  4.  Once  quantified,  direct 


coordination  with  the  respective  resource 
providers  should  generate  sufficient  information 
to  answer  the  question. 

Can  the  joint  test  and  evaluation  be 
completed  in  three  years?  The  test  concept 
and  approaches  for  achieving  the  JT&E 
objectives  identified  the  test  activities  that  are 
required  to  resolve  the  issues.  Many  of  these 
activities  will  be  conducted  in  conjunction  with 
already  scheduled  Joint  Staff  or  Service  exercises 
or  events,  some  of  which  may  be  conducted  in 
OSD  or  Service  facilities.  Coordination  with  the 
organizations  that  control  these  exercises,  events, 
ranges,  or  facilities  will  provide  the  information 
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required  to  determine  if  JT&E  participation  is 
possible.  This  information  will  be  incorporated 
into  the  T&E  schedule  that  will,  in  turn,  provide 
information  to  answer  the  question. 

It  may  not  be  possible  to  complete  a  JT&E 
in  the  three-year  time  frame;  it  is  incumbent  upon 
the  FSD  to  plan  an  achievable  JT&E.  If  the 
JT&E  cannot  be  achieved  it  three  years,  the  FSD 
must  own  up  to  that  fact  and  fully  justify  the 
required  time  frame.  The  FSD  should  not  base 
the  JT&E  on  the  expectation  that  additional  time 
and  funds  will  be  made  available  once  the  JT&E 
is  chartered. 

Are  the  benefits  gained  worth  the 
cost?  The  question  of  benefits  gained  versus 
cost  will  be  more  difficult  to  answer.  The 
benefits  gained  will  be  based  on  an  untested 
concept  or  proposed  solution  to  an  issue  or 
problem,  and  the  overall  cost  estimates  will  be 
based  on  the  best  estimates  of  the  support 
required  to  participate  in  future  events.  In  other 
words,  the  data  available  to  answer  this  question 
will  be  subjective,  and  the  credibility  of  this  data 
may  be  challenged.  The  data  gathered  to  answer 
this  question  should,  therefore,  be  validated  in 
terms  of  data  source  and  general  acceptance  by 
the  T&E  community.  The  key  to  answering  this 
question  lies  in  the  source  of  the  data  and  the 
credibility  of  the  source. 

Can  a  solution  be  found  and  do 
technologies  exist  to  support  the 
solution?  The  joint  test  concept  is  a  proposed 
solution  to  the  nominated  concepts  or  issues. 
The  development  of  the  test  method  matrix 
involved  examination  of  available  test  methods  in 
terms  of  their  credibility,  the  ability  of  producing 
the  requisite  data  and  its  fidelity,  and  the  general 


acceptance  of  the  test  results.  The  JFS  must 
develop  sufficient  information  to  judge  the 
probabilities  that  the  technologies  exist  to  support 
the  proposed  solution.  If  a  proposed  solution 
cannot  be  found  or  an  acceptable  test  method 
(technology)  is  not  available,  the  FSD  should 
immediately  advise  the  DDT&E  of  the  situation  to 
discuss  alternative  solutions. 

Is  joint  testing  the  best  vehicle  for 
resolving  the  nominated  concepts  or 
issues?  Answering  this  final  question  will  be 
based  primarily  on  subjective  inputs.  The  JFS 
team  should  solicit  inputs  from  experts  on  the 
subject  at  the  highest  level  possible.  If  the 
nomination  involves  two  or  more  Services,  there 
are  two  vehicles  for  resolving  the  nomination:  a 
multi-Service  operational  test  and  evaluation 
(MOT&E)  or  a  JT&E.  If  it  does  not  involve 
policy,  doctrine,  or  Service  parochialism,  a 
MOT&E  is  probably  the  best  test  vehicle.  If 
policy,  doctrine,  new  concepts,  or  Service 
parochialism  are  involved  or  even  suspected, 
then  a  JT&E  is  probably  the  best  vehicle.  Other 
considerations  in  answering  this  question  could 
include  the  time  required  to  develop,  test,  and 
implement  a  solution;  the  magnitude  of  the 
concepts,  issues,  and  objectives;  and  cost. 

Environmental  Considerations 

As  the  specific  test  activities  are  defined  in 
the  JT&E  concept,  the  JFS  must  include 
consideration  of  environmental  factors  and  their 
potential  impact  on  the  feasibility  question. 
Government  regulations  mandate  that  DoD 
activities  comply  with  all  legal  statutes,  such  as 
the  National  Environment  Policy  Act  (NEPA),  to 
ensure  the  preservation  of  natural  resources. 
There  may  be  issues  associated  with  a  proposed 
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JT&E  that  require  mitigating  actions  or  schedule 
changes  to  avoid  an  adverse  environmental 
impact.  Because  such  actions  may  involve 
lengthy  procedures,  such  as  Environment 
Assessments  (EA)  or  Environmental  Impact 
Statements  (EIS),  the  JFS  should  anticipate  and 
avoid  environmental  issues  whenever  possible. 
For  assistance  on  environmental  issues,  the  FSD 
should  contact  the  lead  Service  JT&E  POC. 
Annex  K  contains  a  more  detailed  discussion  of 
environmental  compliance  requirements. 

Security  Considerations 

The  JFS  must  also  consider  security 
requirements  and  how  they  could  affect 
feasibility.  Risk  assessments  regarding  the 
system  under  test  may  be  required.  Operations 
security  (OPSEC)  for  all  phases  of  the  JT&E 
could  be  a  major  determinant  as  to  the  location 
and  timing  of  test  events.  Additionally,  the  JFS 
timelines  must  consider  the  time  that  may  be 
required  to  acquire  equipment  for  processing 
classified  information  as  well  as  personnel 
clearances  and  upgrades.  The  requirement  for  a 
SCIF  and  SCI  billets  can  significantly  impact  the 
funds  and  time  needed  to  conduct  a  JT&E. 

Other  Considerations 

There  are  a  number  of  other  variables  over 
which  the  FSD  has  little  or  no  control,  but  which 
could  influence  feasibility.  One  is  the  level  of 
JT&E  program  funding.  While  a  JT&E  may  be 
feasible  and  project  significant  payoffs,  there 
may  be  insufficient  funding  available  for  charter. 
Another  variable  involves  political  interest  and 
sensitivities.  Congressional  interest,  both 
favorable  and  unfavorable,  can  have  significant 
influence  over  which  JT&Es  are  chartered.  Other 


variables  could  include  the  availability  and 
priority  for  use  of  test  assets  and  resources 
(equipment,  instrumentation,  facilities,  ranges, 
manpower,  and  technology)  that  are  required  for 
the  proposed  JT&E  and  Service  support 
(willingness  to  provide  funding,  personnel,  and 
equipment)  for  the  JT&E.  The  FSD  must  remain 
sensitive  to  these  variables  and  consider  each  in 
making  conclusions  and  recommendations 
regarding  the  feasibility  of  the  nominated  JT&E. 

PART  TWO 

ANALYSIS  PLAN  FOR  ASSESSMENT 

The  Analysis  Plan  for  Assessment  (APA)  is 
a  continuation  of  the  sequential  test  concept 
process  that  defines  and  formalizes  what  actions 
and  resources  are  required  to  accomplish  the 
JT&E.  In  the  previous  part  of  this  chapter,  the 
JFS  addressed  whether  a  JT&E  is  necessary  and 
feasible.  If  a  JT&E  is  justified,  the  JFS  should 
develop  a  APA  to  formalize  what  will  be 
required  to  accomplish  the  JT&E,  who  is 
responsible  for  the  required  tasks,  when  and 
where  the  test  activities  will  be  conducted,  and 
what  resources  will  be  required.  The  specifics 
of  how  each  test  task  and  activity  will  be 
conducted  is  covered  in  a  test  plan  for  that 
specific  test  activity,  which  will  be  developed  by 
a  JTF  when  chartered.  The  APA  is  a  formal 
stand-alone  document  that  is  presented  with  the 
JFS  final  report. 

N  .  REFINE  AND  INCORPORATE 
THE  JT&E  CONCEPT  INTO  THE 
APA 

The  JFS  team  should  be  continually 
refining  and  expanding  the  level  of  detail  in  the 
approved  JT&E  concept  and  quantifying  the  OSD 
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and  Service  resources  required  to  execute  the 
proposed  JT&E.  When  the  FSD  is  satisfied  that 
the  details  in  the  concept  are  sufficient,  the  JFS 
team  will  formalize  the  JT&E  concept  in  the  form 
of  the  APA. 

JT&E  nominations  involve  concepts  and 
issues  that  vary  in  technical  complexity  and  test 
activities  ranging  from  isolated  studies  to  large 
scale  field  tests  involving  multiple  test  methods, 
locations,  ranges,  force  sizes,  operational 
missions,  and  Service  involvements.  It  is  not 
likely  that  any  APA  structure  will  be  applicable  to 
all  proposed  JT&Es.  A  suggested  APA  outline  is 
included  as  Part  One  of  Annex  N.  The  suggested 
outline  is  provided  as  a  guide  for  those  who  have 
not  previously  developed  a  test  design  and  is  not 
intended  to  restrict  creativity.  The  APA,  when 
formalized,  becomes  the  program  roadmap  for 
the  execution  of  the  JT&E  by  a  JTF. 

The  APA  outlines  what  must  be  done  to 
complete  the  proposed  JT&E  within  a  three-year 
time  frame.  It  also  identifies  where  the  test 
activities  will  be  conducted,  the  relationships 
between  test  events,  the  resources  that  will  be 
required  to  accomplish  the  JT&E,  and  who  is 
responsible  for  providing  the  resources.  The 
APA  will  be  coordinated  through  the  appropriate 
OSD,  Joint  Staff,  and  Service  organizations.  In 
that  the  TAB  reviewed  and  concurred  with  the 
program  test  concept,  the  final  TAB  review  of  the 
APA  will  be  conducted  in  conjunction  with  the 
JFS  final  report  review  at  the  third  JFS 
milestone.  The  DDT&E  will  be  the  approving 
authority  for  the  APA. 


O.  CONSIDER  ALTERNATIVES 

The  JFS  should  consider  alternatives  to  the 
full  scale  test  concept  that  was  previously 
developed  under  few  or  no  constraints  (primarily 
need,  feasibility,  and  time).  As  the  T&E  concept 
definition  matures,  the  FSD  may  find  that 
resource  requirements  increase.  Increases  in 
resource  requirements  and  countervailing  DoD 
budget  limitations  could  result  in  a  feasible  JT&E 
not  being  chartered.  The  FSD  should,  therefore, 
consider  alternatives  for  conducting  the  JT&E 
that  are  based  on  reduced  funding  levels,  and  that 
would  nonetheless  provide  meaningful  results. 
If  the  program  is  technically  feasible,  there  may 
be  alternatives  that  could  resolve  the  issues,  or  at 
least  the  critical  issues.  The  FSD  should 
anticipate  and  identify  an  expected  level  of 
funding  and  develop  a  baseline  program  below 
which  few  or  no  benefits  would  occur.  The  FSD 
then  could  consider  alternatives  that  are  reduced 
but  above  this  baseline.  Alternatives  should  be 
based  on  consideration  of  reduced  resource 
requirements  made  possible  by  changing  test 
conditions  (e.g.,  field  exercise  vice  lab  test), 
using  a  more  bare-bones  tactical  scenario, 
increased  use  of  historical  data,  and  so  on.  Two 
basic  techniques  that  have  been  used  by  previous 
feasibility  studies  for  the  downward  rescoping  of 
JT&E  test  concepts  are  programmatic 
restructuring  and  technical  restructuring. 

Programmatic  restructuring  involves 
tradeoffs  between  resources,  time,  funds,  and  the 
benefits  to  be  gained  from  the  JT&E  results.  A 
reduced  program  might  be  able  to  resolve 
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the  JT&E  issues  for  only  a  limited  set  of 
circumstances.  These  circumstances,  however, 
might  be  those  that  are  most  critical  or  those  that 
occur  most  frequently.  Programmatic 
restructuring  might  also  be  achieved  by 
dissecting  the  joint  .test  issues  and  addressing 
only  those  considered  most  significant.  Further 
restructuring  could  consider  reducing  the  number 
of  controlled  test  variables  (environments, 
systems,  tasks)  in  the  test  cell  matrix. 

Technical  restructuring  involves 
tradeoffs  between  the  cost  and  risk  associated 
with  each  of  the  planned  test  activities.  For 
example,  the  results  from  dedicated  field  tests  are 
credible,  but  are  the  most  expensive  form  of 
testing.  On  the  other  hand,  simulations  can  be 
less  expensive,  but  have  been  less  credible  to  the 
T&E  community.  Most  JT&Es  will  involve  a 
mix  of  several  test  methods  to  achieve  the 
planned  test  activities.  Restructuring  the  test 
method  matrix  (see  Figure  3-6)  could  produce 
reduced  resource  requirements  and,  in  turn, 
reduced  funding  requirements. 

PART  THREE 

REVIEWS,  BRIEFINGS,  &  REPORTS 
P.  PROGRESS  REVIEWS 

The  primary  recipient  of  JFS  progress 
reviews  will  be  the  DDT&E.  The  FSD  should 
establish  a  schedule  to  provide  quarterly 
update  briefings.  These  briefings  will  review 
the  general  and  financial  status  of  the  JFS  and 
present  topics  of  current  interest  to  the  DDT&E, 
such  as  critical  milestones,  issues,  status  of 
document  preparation,  and  results  of 
coordination.  As  needed,  a  detailed  financial 
profile,  including  commitments,  expenditure, 


remaining  balance,  and  spending  plans,  should 
be  provided.  It  is  very  important  to  keep  the 
Services  informed  and  to  address  their  concerns 
through  proactive,  routine  IPRs.  Other  agencies 
may  be  included  in  periodic  but  less  frequent 
reviews. 

In  preparing  for  these  reviews,  the  FSD 
should  develop  modular  briefings  using  a 
standard  format.  Use  of  modular  templates  on  a 
LAN  will  facilitate  the  development  and  upgrade 
of  these  briefings.  Contributors  to  the  briefings 
should  update  the  data  in  their  respective  sections 
at  specific  times  each  month.  The  briefings  can 
also  be  used  for  internal  status  reviews,  popup 
responses  to  information  or  data  queries  and 
updates  to  other  agencies  as  the  need  arises.  The 
primary  thrust  of  these  briefings,  however,  is  to 
support  the  DDT&E  quarterly  reviews,  and 
inputs  to  the  briefings  should  be  timed  and 
formatted  to  be  compatible  with  that  requirement. 
The  basic  elements  in  the  reviews  include  JFS 
progress,  budget,  and  problems/solutions. 

JFS  Progress 

This  area  of  the  review  includes  JFS 
milestone  completion,  projections,  slippage, 
problems,  challenges,  and  risks.  Items  of 
particular  importance  are  completed  and  projected 
activities,  decision  points,  and  interfaces  with 
other  agencies.  The  identification  of  long-lead 
preparations  and  projected  procurement 
requirements  should  also  be  noted. 

Budget 

A  graphic  depiction  of  the  status  of  JFS 
monies  should  be  presented  that  displays  funds 
committed,  obligated,  and  expended  and  the 
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remaining  balance.  The  planned  versus  actual 
cost  and  the  cost  of  future  JFS  activities  should 
be  presented  when  significant.  Direct  cite 
monies,  reimbursable  and  Service  funds,  and 
contracts  should  be  addressed. 

Problems  and  Solutions 

This  area  of  the  review  should  remain 
flexible  because  it  is  situational-dependent. 
Nevertheless,  it  is  important  to  keep  the  DDT&E 
informed  of  problem  areas  encountered, 
solutions  worked,  those  remaining  unresolved, 
assistance  needed,  and  the  potential  impact  of  the 
problems  on  the  JFS  or  proposed  JT&E. 
Applicable  areas  in  which  DDT&E  awareness  is 
necessary  include  coordination  of  JFS  and 
projected  JT&E  requirements  within  OSD,  the 
releasability  of  classified  information,  resource 
priorities,  and  OSD/Service  coordination  relative 
to  participation  in  exercises,  tests,  or  the  support 
of  dedicated  JT&E  activities.  DDT&E  must 
approve  any  press  statements  or 
releases. 

Q.  JOINT  FEASIBILITY  STUDY 

STATUS  REPORTS 

In  addition  to  the  progress  reviews,  the 
FSD  will  provide  a  monthly  status/activity  report 
to  the  DDT&E.  These  reports  are  not  intended  to 
duplicate  the  information  in  the  progress  reviews, 
but  rather  keep  OSD  informed  about  how  the  JFS 
is  faring,  what  the  problems  are,  where  OSD  and 
Service  help  is  needed,  and  what  are  the  JFS 
recommended  courses  of  action.  The  FSD 
should  ask  himself/herself  what  would  he/she 
want  to  know  to  help  manage  the  feasibility  study 
and  report  that  information.  Because  of  the 
environment  in  which  we  work,  the  report  must 
also  address  the  current  financial  status.  Funds 


should  be  delineated  by  source  and  year,  amount 
received,  committed,  obligated,  expended  and  the 
remaining  available  balance.  The  reports  should 
not  be  extremely  detailed  nor  time  consuming  for 
either  the  writer  or  the  reader.  Items  to  be 
considered  for  inclusion  are: 

•  Number  of  personnel  assigned. 

•  Comment  on  adequacy  of  support  and  OSD 
assistance  needed. 

•  Significant  visits  to  the  JFS  during  the 
reporting  period  and  projected  visits  during 
the  upcoming  period. 

•  Schedule  of  upcoming  activities. 

•  Highlighted  issues  related  to  Service 
coordination,  resources,  etc. 

•  Accomplishments. 

•  Problems. 

R.  JFS  FINAL  REPORT 

The  JFS  final  report  is  the  most  important 
document  that  the  JSD  will  produce.  It  is  the 
documentation  upon  which  the  D,T,SE&E  will 
base  his  or  her  decision  relative  to  chartering  of 
the  JT&E  program.  It  is  incumbent  on  the  FSD  to 
devote  the  required  time  and  resources  to  assure 
that  the  final  report  and  its  accompanying 
briefings  are  quality  products  and  that  they 
contain  meaningful  results,  conclusions,  and 
recommendations.  Further,  these  reports  and 
briefings  must  be  supported  by  precise  and 
reasonable  documentation. 

The  APA  is  an  integral  part  of  the  JFS  final 
report,  and  both  should  be  fully  coordinated  with 
OSD,  Joint  Staff,  and  the  Services  before 
presentation  to  the  TAB  for  review.  The  JFS 
final  report  should  discuss  the  necessity  and 
feasibility  of  the  proposed  nomination,  its  scope, 
concept,  methodologies,  limitations,  risks,  and 
resource  requirements.  If  the  JFS  recommends 
the  chartering  of  a  JT&E  the  JFS  final  report 
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should  also  include  a  recommendation  for  the 
location  and  staffing  of  the  JTF.  The 
recommendation  for  JTF  location  should  be 
based  on  considerations  of  available  facilities, 
travel  requirements  relative  to  projected  test 
activities,  and  the  availability  of  qualified 
personnel  to  support  JT&E  planning  and 
execution  requirements.  A  recommended  format 
for  the  JFS  final  report  is  included  in  Annex  N. 

S  .  JFS  MANAGEMENT  REPORT 

The  JFS  management  report  serves  three 
purposes.  First,  it  provides  the  DDT&E  with  the 
FSD's  assessment  of  the  JFS,  to  include 
accomplishment  of  the  directed  mission.  Second, 
it  documents  lessons  learned  for  consideration  in 
the  organization  and  management  of  future  JFSs. 
Documenting  lessons  learned  has 
historically  been  one  of  the  poorly 
supported  parts  of  the  JFS  process. 


Admitting  a  lesson  learned  is  frequently 
interpreted  as  admitting  a  mistake.  Lessons 
learned  seldom  benefit  the  author  and,  thus, 
frequently  receive  a  minimal  level  of  effort. 
However,  lessons  learned  can  be  of  significant 
value  to  the  effectiveness  of  future  JFSs.  In  the 
preparation  of  the  lessons  learned  and 
recommended  actions,  the  FSD  should  separate 
those  that  are  systemic  from  those  that  are  new  or 
unique  so  as  to  add  emphasis  to  both  types. 
Third,  the  report  provides  recommendations  for 
actions  to  improve  the  efficiency  and 
effectiveness  of  future  JFSs.  Feedback  on 
problems  and  solutions  tried  (successful  or  not) 
from  previous  JFSs  should  be  included. 

If  the  JFS  is  not  chosen  to  continue  as  a 
JT&E,  a  management  report  should  be  completed 
within  30  days  following  the  non-chartering 
decision.  A  suggested  format  for  the  JFS 
management  report  is  included  as  Annex  N. 
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CHAPTER  4 

RESOURCE  REQUIREMENTS 


A.  INTRODUCTION 

The  JFS  must  address  resource 
requirements  at  two  levels:  1)  to  support  its  own 
daily  operation  and  execution  of  its  chartered 
mission  and  2)  to  project  JT&E  resource 
requirements  for  the  JTF  chartering  decision. 
Resource  requirements  in  support  of  the  JFS  are 
discussed  in  detail  in  Section  C  of  Chapter  2. 

Decisions  relative  to  the  chartering  of  a  JTF 
will  be  based  on  information  and  data  collected 
during  the  JFS,  and  the  resulting 
recommendations  will  be  briefed  during  the 
coordination  process  and  published  in  the  JFS 
final  report.  A  major  factor  contributing  to  the 
"is  it  feasible?"  question  and,  thus,  the  chartering 
decision,  is  the  availability  of  resources  to 
accomplish  the  activities  as  outlined  in  the 
proposed  APA.  The  JFS  team,  in  conjunction 
with  the  supporting  Services,  will  develop  a 
Consolidated  Resource  Estimate  (CRE)  that 
identifies  the  funding,  personnel,  and  equipment 
required  to  execute  the  proposed  JT&E.  In  the 
past,  the  reliability  of  the  data  in  the  CRE  has  had 
a  direct  bearing  on  the  credibility  of  the  JFS 
recommendations. 

B .  CONSOLIDATED  RESOURCE 
ESTIMATE 

Request  for  Service  Resources 

Most,  if  not  all,  of  the  personnel, 
equipment,  and  facilities  required  to  support  the 


testing  efforts  of  the  JT&E,  as  well  as  the 
support  of  the  JTF  cadre,  will  be  provided  by  the 
Services.  Thus,  the  Services  must  include  these 
requirements  in  their  resource  management 
planning  documents  and  budget  requests  in  order 
to  ensure  that  they  will  be  available  to  meet  the 
JTF  needs.  The  JFS,  in  coordination  with  the 
lead  and  supporting  Services,  will  develop  an 
estimate  of  the  resources  (personnel,  equipment, 
and  funds)  required  along  with  an  expected  time 
frame  and  suggested  source  to  fill  the  needs. 
These  estimates  will  be  consolidated  into  a  single 
document,  the  CRE.  An  outline  for  the  CRE  is 
in  Annex  I. 

The  Army  and  Air  Force  have  organizations 
that  support  and  monitor  their  Service's 
involvement  in  JTFs  and  have  established  formal 
processes  to  review,  manage,  and  provide 
resources  for  JT&E  support.  On  the  other  hand, 
the  Navy  and  Marines  assign  staff  responsibility 
for  JT&E  support  on  a  case-by-case  basis  along 
operational  staff  functional  lines  of  interest. 
When  support  for  a  particular  JT&E  is  approved, 
a  Navy  or  Marine  staff  sponsor  is  assigned  to 
coordinate  and  provide  the  required  resources. 
No  matter  what  form  the  Service  JT&E  support 
agencies  take,  it  should  be  noted  that  the 
processes  for  recognizing,  coordinating,  and 
providing  the  required  resource  support  can  take 
in  excess  of  a  year.  A  JT&E  reality  is  that 
Service  resources  may  not  be  available  for  the 
first  18  months  of  a  36-month  test  and 
evaluation.  It  is  imperative  that  JT&E 
resource  requirements  be  identified  and 
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submitted  to  the  Services  in  the  proper 
formats  as  soon  as  possible  after  the  JTF 
is  chartered . 

If  it  is  apparent  that  the  JFS 
recommendation  will  be  to  charter  a  JT&E  and 
that  decision  is  fully  supported  by  the  Services, 
the  JFS  Service  deputies  should  break  out  their 
portion  of  the  CRE  in  the  format  of  their 
particular  Service  and  have  these  breakouts 
available  for  JTF  action  upon  chartering.  The 
JFS  Service  deputies  should  coordinate  these 
resource  requirements  with  their  respective 
Services  so  they  can  anticipate  the  requirements 
in  order  to  compress  the  lead  times  for 
acquisition  of  the  resources.  This  will  provide 
for  the  earliest  possible  establishment  of  JT&E 
resource  requirements  in  Service  channels. 

United  States  Army 

The  Operational  Test  and  Evaluation 
Command  (OPTEC),  in  coordination  with  the 
Deputy  Chief  of  Staff  for  Operations  (DCSOPS), 
is  the  overall  Army  manager  and  resource 
provider  for  JT&E  support.  OPTEC  has 
established  an  interdepartmental  Army 
committee,  the  Test  Schedule  and  Review 
Committee  (TSARC),  that  provides  centralized 
management  of  Army  resources  to  support  test 
activities.  The  formal  document  for  submitting 
requests  for  Army  resources  to  the  TSARC  is  the 
Outline  Test  Plan  (OTP).  New  or  revised  OTPs 
are  forwarded  to  OPTEC  for  review  and 
coordination  prior  to  presentation  to  the  TSARC 
for  their  consideration. 

TSARC  meets  bi-annually  to  review  and 
approve  or  disapprove  requests  for  utilization  of 
Army  assets  in  support  of  test  activities.  The 


purpose  of  the  TSARC  is  to  maximize  the  use  of 
limited  resources  and  minimize  the  impact  of 
these  activities  on  operational  units  and  their 
readiness.  OPTEC  publishes  the  TSARC 
Handbook  that  contains  the  guidelines  and 
procedures  for  requesting  Army  resources. 
Copies  of  the  TSARC  Handbook  can  be  obtained 
through  OPTEC  CSTE-OPM,  telephone  (703) 
756-1516.  TSARC  responsibilities  include: 

•  Reviewing  and  recommending  coordinated 
OTPs  for  inclusion  in  the  Five  Year  Test 
Plan  (FYTP). 

•  Ensuring  adherence  to  the  one-year 
notification  requirement  for  resource 
commitments  or  enforcing  compliance  with 
the  exception  policy. 

•  Reviewing  and  recommending  approval  of 
T&E  priorities. 

•  Reviewing  and  coordinating  resources  to 
support  T&Es. 

•  Resolving  conflicts  between  user  test 
requirements  and  other  missions. 

OTPs  are  submitted  in  March  and 
September  of  each  year  using  the  format 
contained  in  Appendix  A  to  the  TSARC 
Handbook.  They  should  contain  requisite 
information  on  support  requirements  available  at 
the  time  of  submission.  Once  submitted,  OTPs 
are  updated  through  the  TSARC  process  in 
March  and  September  of  each  year  as  revised 
OTPs.  OTPs  are  to  be  submitted  for 
TSARC  consideration  at  least  one  year 
before  the  first  resource  is  required  to 
allow  resource  planning  against 
approved  OTPs.  If  the  Army  resources  are  to 
be  provided  by  the  Major  Command  (MACOM) 
or  Field  Operating  Agency  (FOA)  responsible  for 
the  OTP,  the  OTP  may  be  submitted  one  TSARC 
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cycle  (i.e.,  six  months)  before  the  scheduled  test 
date.  This  will  ensure  that  the  required  resources 
are  entered  into  the  FYTP  for  at  least  one  TSARC 
cycle  in  order  to  provide  for  the  monitoring  of 
test  resources  and  test  status.  OTPs  that  do  not 
provide  at  least  one  year  between  TSARC 
approval  and  first  resource  commitment  will  be 
processed  as  an  exception  to  policy.  Exception- 
to-policy  OTPs  will  be  forwarded  to  the  TSARC 
under  a  cover  letter  signed  by  a  general  officer  of 
a  TSARC  member  headquarters,  stating 
justification  for  the  requested  policy  exception. 

United  States  Air  Force 

Air  Force  Instruction  99-106,  Joint  Test 
and  Evaluation,  18  March  94,  provides 
guidelines  and  procedures  for  Air  Force 
participation  or  support  of  JT&E  programs 
directed  by  OSD.  It  describes  the 
responsibilities,  planning,  and  execution  for 
JT&E  participants.  HQ  USAF/TE  sets  Air 
Force  policy  for  JT&Es  and  is  the  office  of 
primary  responsibility  (OPR)  for  Air  Force 
participation  in  the  JT&E  program.  The  Air 
Force  Operational  Test  and  Evaluation  Center 
(AFOTEC)  assists  HQ  USAF/TE  in  the 
execution  of  the  JT&E  program  and  is  designated 
as  the  Air  Force  manager  for  chartered  JT&Es. 
All  requests  for  JT&E  dedicated  manpower 
support  are  forwarded  to  AFOTEC  for 
processing  and  HQ  USAF/TE  validation. 
AFOTEC/XRJ  is  the  AFOTEC  JT&E  action 
office  and  receives  specific  JT&E  support 
requests.  The  formal  document  for  requesting 
resources  from  the  Air  Force  to  support  joint 
tests  is  the  Test  Resource  Plan  (TRP).  Air  Force 
MAJCOMs  and  FOAs  will  provide  most,  if  not 
all,  of  the  required  resources.  The  guidelines  and 
procedures  for  requesting  Air  Force  resources  for 


a  JT&E  are  contained  in  AFOTECI  99-101  and 
AFOTEC  Pamphlet  55-8.  Copies  of  this 
instruction  and  pamphlet  can  be  obtained  by 
contacting  AFOTEC/XRJ,  telephone  (505)  846- 
5339. 

AFOTEC/XRJ  is  responsible  for: 

•  Assisting  in  the  preparation  of  TRPs. 

•  Monitoring  Air  Force  support  of  JT&Es 
through  the  TRP  process. 

MAJCOMS  and  FOAs  provide  most  of 
the  resources  and  personnel  to  support  JT&Es. 
Each  MAJCOM  and  FOA  will  maintain  a 
permanent  POC  for  JT&E  matters.  Information 
relative  to  these  POCs  can  be  obtained  from 
AFOTEC/XRJ.  MAJCOMs  and  FOAs  are 
responsible  for: 

•  Supporting  AFOTEC  in  interfacing  with  the 
JT&E  directors. 

•  Providing  personnel  and  equipment  as 
tasked  in  Air  Force  test  directives. 

•  Programming  resources  against  approved 
TRPs. 

•  Evaluating  JT&E  impacts  on  operational 
capabilities. 

The  TRP  is  the  vehicle  for  integrating 
JT&E  resource  requirements  into  the  Air  Force 
resource  planning,  programming,  budgeting,  and 
execution  system.  The  guidelines  and 
procedures  for  preparing  and  submitting  the  TRP 
are  contained  in  AFOTEC  Pamphlet  55-8.  New 
TRPs  will  be  coordinated  with  the  appropriate 
commands  and  agencies  and  forwarded  to 
AFOTEC/RM  to  coincide  with  the  two 
established  TRP  cycles,  October  through 
December  and  April  through  June.  Out-of-cycle 
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TRPs  may  be  submitted  if  major  program 
restructuring  or  other  substantial  changes  occur. 
TRPs  will  be  kept  current  to  reflect  maturing 
JT&E  resource  requirements.  TRPs  should  be 
revised  as  soon  as  program  or  test  schedules 
change  and  should  always  reflect  the  most 
current  status  of  resource  requirements. 

United  States  Navy 

The  principal  POC  for  JT&Es  is  located  in 
the  Office  of  the  Director  of  Test  and  Evaluation 
and  Technology  Requirements  (N091), 
specifically  in  the  T&E  Division  (N912D2).  The 
JT&E  POC  who  provides  liaison  and  interface 
with  OSD  on  JT&E  policy  and  resources  issues 
is  N912D2.  In  contrast  to  the  Air  Force  and 
Army,  the  Navy  selects  and  delegates  authority 
and  responsibility  for  JT&Es  on  a  case -by-case 
basis. 

For  Navy  support  of  a  specific  JT&E, 
N091  will  be  responsible  to  staff  the  selection  of 
an  appropriate  sponsor  from  the  Chief  Naval 
Operations  (CNO)  staff  organization  and  will 
assign  a  Test  and  Evaluation  Identification 
Number  (TEIN)  and  a  CNO  priority.  The  CNO 
staff  sponsor  is  responsible  for  staffing  the 
selection  of  a  field-level  Navy  agency  to  support 
the  JT&E.  The  responsibilities  and  staff  actions 
within  the  CNO  staff  for  JT&Es  are  shared 
between  N912D2  and  the  technical  sponsor.  This 
procedure  provides  a  team  approach  in  which  the 
JT&E  POC  is  knowledgeable  concerning  the 
entire  spectrum  of  JT&E  activity.  The  CNO 
sponsor  provides  specific  functional  expertise, 
and  the  field-level  sponsor  supports  the  JFS  or 
JT&E. 


There  are  no  specific  Navy 
directives  relative  to  obtaining  resources 
to  support  a  joint  test.  Resource 
requirements  must  be  requested  and  arranged  for 
on  a  case-by-case  basis  through  the  CNO  and 
field  level  sponsors. 

N091  is  responsible  for  obtaining  and 
assigning  the  TEIN  and  the  CNO  priority  and  for 
staffing  the  selection  of  the  CNO  staff  sponsor. 

The  CNO  staff  sponsor  is  responsible 
for  staffing  the  selection  of  the  field-level 
sponsor  and  for  providing  the  field-level 
structure  for  supporting  JT&Es. 

The  field-level  sponsor  is  responsible 
for  staffing  and  providing  resource  requirements 
for  the  JT&E,  to  include  request  for  manpower 
billets  and  the  requisition  of  personnel. 

The  Operational  Test  and  Evaluation 
Force  (OPTEVFOR)  is  the  independent 
operational  test  agency  for  the  Navy  and  is 
responsible  for  the  operational  test  and  evaluation 
of  weapons  systems,  ships,  aircraft,  equipment, 
procedures,  and  tactics,  when  required. 
OPTEVFOR  is  a  valuable  source  of  testing 
expertise  and  assistance  to  the  JT&E. 

United  States  Marine  Corps 

Within  the  Headquarters  Marine  Corps 
(HQMC)  staff,  the  responsibilities  for  JT&E 
support  are  shared  between  several  deputies.  The 
Deputy  Chief  of  Staff  for  Requirements  and 
Programs  (DC/S  R&P)  is  responsible  to 
coordinate  Program  Objective  Memoranda 
(POM)  activities.  The  Deputy  Chief  of  Staff  for 
Manpower  and  Reserve  Affairs  (DC/S  M&RA), 
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after  consultation  with  Marine  Corps  System 
Command  (MCSYSCOM)  and  the  Director 
Marine  Corps  Operational  Test  and  Evaluation 
Agency  (MCOTEA),  is  responsible  for  the 
validation  of  manpower  and  personnel 
requirements  and  the  identification  of  Service 
representatives  for  approved  JFSs  and  JTFs. 
The  Fiscal  Director  of  the  Marine  Corps  (FDMC) 
is  responsible  to  coordinate  with  the  DC/S  R&P 
to  ensure  that  budgetary  and  programmatic 
decisions  support  the  JT&E,  the  Marine  Corps 
mission,  and  the  budget. 

MCO  5000.1 IB  is  the  Marine  Corps  Order 
(MCO)  governing  test  and  evaluation,  to  include 
participation  in  and  support  of  JT&Es.  This 
MCO  is  being  staffed  for  revision.  The  FSD  can 
obtain  a  current  version  by  contacting  the 
MCSYSCOM  JT&E  POC  at  telephone  (703) 
640-5963. 

Requests  for  Marine  Corps  resources  to 
support  a  JT&E  are  staffed  on  a  case-by-case 
basis.  The  principal  POC  for  a  specific  JT&E  is 
MCSYSCOM,  specifically  the  Test  and 
Evaluation  Division  (MCSYSCOM/T&E). 
MCSYSCOM/T&E  will  coordinate  requests  for 
resources  with  the  appropriate  HQMC  staff  and 
MCOTEA.  The  HQMC  staff  will  coordinate 
those  actions  necessary  to  provide  the  approved 
resource  support.  MCOTEA  will  recommend  to 
the  Commandant  of  the  Marine  Corps  (CMC)  the 
selection  of  a  Fleet  Marine  Force  (FMF)  to 
support  the  joint  testing.  The  FMF  will  designate 
FMF  coordinators  at  applicable  units  and  provide 
the  approved  resources  and  support. 

MCSYSCOM/T&E  is  the  principal  POC 
for  all  JT&E  support  requirements,  to  include 


technical  assistance,  personnel,  equipment,  and 
forces. 

DC/S  R&P,  after  consultation  with 
MCSYSCOM  and  the  Director  MCOTEA,  is 
responsible  for  the  validation  of  manpower  and 
personnel  requirements  and  the  identification  of 
Service  representatives  for  approved  JT&E 
positions. 

MCOTEA  is  the  Marine  Corps 
independent  T&E  organization  responsible  for 
adequate  testing,  objective  evaluation,  and 
independent  reporting  in  support  of  the  Marine 
Corps  acquisition  process.  MCOTEA  is  also 
responsible  to  recommend  to  CMC  the  selection 
of  the  FMF  to  support  joint  testing;  to  provide 
technical  support  to  the  Marine  Corps  joint 
technical  director  in  the  form  of  assistance  in  test 
planning,  direction,  and  reporting;  to  ensure  the 
quality  control  of  Marine  Corps  participation;  and 
to  conduct  independent  evaluations  as  may  be 
required. 

The  Fleet  Marine  Forces  are 
responsible  to  designate  FMF  test  coordinators  in 
the  FMF  headquarters  and  at  each  division  or 
wing  and  to  provide  the  approved  resources 
required  to  conduct  or  support  the  JT&E. 

C .  COST  ESTIMATES 

Cost  estimates  of  the  resources  in  the  CRE 
can  be  developed  by  quantifying  each  in  terms  of 
what  will  be  required,  where  it  will  be 
required,  when  it  will  be  required,  how  long  it 
will  be  required  and  how  it  will  be  provided  at 
the  required  location.  This  can  be  accomplished 
by  decomposing  the  CRE  into  specific  items, 
costing  each,  and  then  aggregating  the  cost  by 
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time  and  responsibility.  The  CRE  decomposition 
should  be  tailored  to  the  unique  requirements  of 
the  JT&E.  The  following  are  suggested  topics 
that  might  be  considered  in  a  breakdown  of  the 
resource  requirements  in  the  CRE. 

•  Category  (personnel,  facility  requirements, 
transportation,  materials,  support 
equipment,  aircraft  sorties,  etc.). 

•  Type  (rank,  technical  qualifications,  type  of 
aircraft,  secure  space,  etc.). 

•  Location  (home,  deployed,  labs,  ranges, 
etc.). 

•  Duration. 

•  Relocation  and  transportation 
requirements. 

While  it  is  realized  that  each  proposed 
JT&E  is  unique  and  may  require  unique  data 
collection  and  analysis  equipment,  there  are  many 
items  and  requirements  that  are  similar  and 
constant  between  JT&Es.  Annex  J  contains  a 
generic  test  program  cost/budget  matrix  that  can 
be  used  to  identify  and  cost  potential  JTF 
requirements.  Because  each  proposed  JTF  will 
have  unique  requirements,  the  matrix  will  have  to 
be  adjusted  and  tailored  by  the  JFS  team  to  meet 
the  needs  of  the  proposed  JT&E. 

The  cost  for  usage  of  range  and  laboratory 
facilities  should  be  obtained  directly  from  the 
facility  operators.  The  cost  associated  with 
Service-provided  resources  could  be  obtained  by 
the  JFS  deputies  through  interface  with  their 
Services'  operational  commands  and  logistical 
agencies.  The  cost  to  satisfy  unique  JT&E 
hardware,  software,  and  instrumentation 
interfaces  or  modifications  should  be  obtained 
from  the  most  reliable  source  available  to  the  JFS 
(OTAs,  labs,  ranges,  vendors,  etc.).  DESA  is  a 


valuable  source  for  cost  information,  as  well  as 
the  actual  provisioning  of  instrumentation,  data 
collections,  and  analysis  equipment.  DESA  has 
both  a  considerable  inventory  of  free  issue  testing 
equipment  and  cost  data  from  previous  JFSs  and 
JT&Es. 

The  costs  of  the  individual  items  are  then 
aggregated  by  location,  time,  and  responsibility 
(OSD  or  the  Services)  and  incorporated  into  an 
overall  cost  estimate  for  the  JT&E.  If  the 
estimated  cost  exceeds  the  fiscal  guidelines 
outlined  for  the  JT&E  by  DDT&E,  the  JFS 
should  consider  programmatic  or  technical 
restructuring  of  the  JT&E  to  reduce  the  estimated 
cost.  While  it  is  recognized  that  the  actual 
responsibility  for  test  item  cost  may  be  negotiated 
by  the  OSD  and  Service  agreement  (Charter) 
establishing  the  JT&E  program,  there  are  general 
guidelines  for  determining  cost  responsibility  for 
the  CRE  items. 

OSD  provides  partial  funding  to  JT&Es  in 
accordance  with  DoD  Manual  7110-1.  These 
OSD  funds  (PE  0605804D)  are  used  to  pay  costs 
incurred  for  the  direction,  supervision,  and 
performance  of  the  JT&E  and  those  that  are 
unique  to  JT&E  needs,  such  as  development  of 
the  test  design  and  supporting  test  plans; 
procurement,  installation  and  operation  of  unique 
instrumentation;  transportation,  travel,  and  per 
diem  cost  for  the  JTF  staff;  modification  of  test 
articles  to  obtain  test  data;  data  collection, 
reduction  and  analysis;  and  test  reporting.  Thus, 
costs  that  are  incurred  solely  as  a  result  of 
conducting  a  JT&E  and  that  cannot  be  regarded 
as  providing  long-term  military  mission-oriented 
benefits  are  normally  funded  by  OSD. 
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The  Services  are  responsible  for  normal 
operational  and  maintenance  (O&M)  expenditures 
in  support  of  the  JT&E,  such  as  civilian  pay, 
travel  (for  other  than  the  JTF  staff),  utilities  and 
rents,  Service-owned  equipment  maintenance, 
supplies,  printing,  reproduction,  and 
communications.  The  Services  also  provide  the 
support  budgets  and  funds  for  normal  training 
missions,  OPTEMPO,  non-industrial  funded 
aircraft  operations,  and  TDY  costs  for  Service 
exercise  participants  and  personnel  other  than  the 
JT&E  staff  who  are  involved  in  test  activities  to 
collect  data. 

In  general,  OSD  funds  test  unique  costs, 
while  the  Services  fund  administrative,  civilian 
salary,  and  select  other  expenses.  Figure  4-1 
summarizes  normal  funding  responsibilities. 

D .  OTHER  RESOURCE- 

RELATED  JTF  REQUIREMENTS 

Once  the  JT&E  resource  requirements  have 
been  identified  and  quantified,  the  JFS  should 
consider  and  provide  recommendations  to  the 
SAC  and  the  D,T,SE&E  on  manning  and  JTF 
location.  The  JFS  should  also  provide  the  newly 
appointed  Joint  Test  Director  (JTD)  with 
recommendations  relative  to  staffing  and 
equipment  acquisition  actions  that  should  be 
initiated  immediately  after  the  JTF  is  chartered. 

Location 

In  general,  the  recommended  location  will 
be  at  a  military  installation.  Location 
recommendation  should  include: 

•  Availability  of  JTF  office/work  area 
facilities. 


•  Proximity  of  Service  support  agencies  and 
equipment. 

•  Proximity  of  required  test  facilities  (such  as 
labs  and  ranges). 

•  Travel  requirements. 

•  Access  to  communications. 

•  Access  to  special  security  facilities  (if 
required). 

•  Transition  of  JFS  personnel  and  assets  to  the 
JTF. 

•  Availability  of  qualified  support  personnel. 

•  Operating  costs. 

•  Availability  of  housing  and  personnel 
support  facilities. 

•  Quality  of  life  considerations. 

Facilities 

It  is  unlikely  that  the  facilities  used  by  the 
JFS  would  be  adequate  to  house  the  JTF  when 
fully  staffed.  The  JFS  facility,  however,  should 
be  adequate  to  house  the  initial  JTF  cadre  and, 
thus,  provide  the  JTD  time  to  negotiate  for 
additional  space.  The  FSD  should  discuss  JTF 
facility  requirements  with  the  host  installation  and 
should  include  the  availability  of  facilities  to  fully 
support  JTF  requirements  in  his  JTF  location 
recommendation.  Facility  considerations  should 
be  based  on  150-200  square  feet  of  office  space 
for  each  JTF  member  with  an  additional  500- 
1000  square  feet  for  a  briefing  area.  All  of  the 
space  should  be  available  at  a  single  location. 
Facility  considerations  should  also  include  the 
requirement  for  related  computer/ADP  support 
and  for  the  storage  and  handling  of  classified 
documents. 
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CATEGORY 

EXPENSES 

1.  JTD  and  Staff 

A.  Travel 

B.  Per  Diem 

C.  Civilian/Military  Pay 

D.  Personal  Computers 

E.  Office  Copier 

F.  Office  Equipment 

G.  Supplies 

H.  Office  Facilities 

I.  Technical  Advisors  (Tab/Tag)  (TDY) 

J.  Other  Training -Tech,  Staff  (TDY) 

K.  Reports,  Printing,  and  Production 

2.  Contractor 

A.  Test  Planning,  Tech  Support 

B.  Simulation/Model 

3.  Training 

A.  Training  for  Data  Collectors  and 

White  Controllers 

B.  Test  Unique  Doctrine  and  Tactics 
Training  for  Red  Players 

C.  Tactics  Training  for  Blue  Players 

4.  Facilities 

A.  Buildings-Construction,  Lease, 
Alteration 

B.  Utilities 

OSD  SERVICES 


X 

X 

X 

X(l)(4) 

X(4) 

X 


5.  Test  Support 

A.  Hardware/ADP 

X 

And  Resources 

B.  Range  Costs 

X(2) 

C.  Instrumentation  (Unique) 

X(2) 

D.  Surrogates 

X 

E.  Site  Preparation 

X 

F.  Base  Operating  Support 

X 

G.  Player  Communications  Support 

X 

H.  Player  Vehicle  Leasing 

X 

I.  Photo/Visual  Support 

II  NEGOTIABLE! 

J.  OPSEC  Survey/Risk  Management 

X 

K.  Environmental  Studies 

X 

L.  Player  Supplies 

X 

M.  Red/Blue  Force  Travel/Per  Diem 

X 

N.  White  Force  Travel/Per  Diem 

X 

O.  Test  Vehicle  Modification  and 

X 

Transportation 

P.  Surrogate  Modification  and 

X 

Transportation 

Q.  Flying  Hours 

X 

R.  Fuel 

X 

S.  Parts  And  Transportation 

X 

T.  Munitions 

X 

U.  Transport  Of  Munitions 

X 

V.  Data  Collection,  Analysis,  Store 

X 

W.  Industrial  Fund  Flight  Hours 

X 

X.  Simulation  Costs,  Modifications, 

Transportation 

X(3) 

1.  Existing  and  suitable  Service  resources,  facilities,  etc.  that  can  be  made  available  will  be 

provided;  however,  additional  costs  for  JT&E  program  requirements  and  resources 

should  be  negotiated  with  OSD. 

2.  Existing  and  suitable  Service  range/test  facilities  that  can 

be  made  available  will  be 

provided;  however,  any  reimbursement  costs  will  be  funded  by  OSD. 

3.  Existing  and  suitable  Service  simulation  capabilities  that  can  be  made  available  will  be 

provided;  however,  other  additional  costs  including  modification,  transportation,  and 

maintenance  will  be  funded  by  OSD. 

4.  DESA  may  be  able  to  provide;  otherwise,  the  lead  Service  provides. 
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Staffing 

Estimated  manning  requirements  will  be  a 
major  factor  in  the  JTF  chartering  decision.  The 
size  of  a  JTF  will  be  based  on  the  mission  and 
tasks  to  be  accomplished.  JTF  manning  will 
primarily  involve  PCS  personnel  that  are 
provided  through  the  Service  personnel  systems. 
The  Service  personnel  systems  can  take  up  to  10- 
14  months  to  provide  PCS  personnel;  thus,  the 
JTF  could  operate  for  roughly  half  of  its  36- 
month  duration  before  full  staffing  is  achieved. 
It  is,  therefore,  incumbent  on  the  FSD  to  estimate 
JTF  manning  requirements  as  accurately  as 
possible  and  coordinate  these  requirements  with 
the  Services  so  that  no  time  is  lost  in  establishing 
personnel  requirements  once  the  JTF  is 
chartered. 

When  the  SAC  recommends  approval  of 
the  JTF  charter,  each  Service  member  of  the  SAC 
commits  his  Service  to  provide  the  appropriate 
resources  to  complete  the  JT&E.  Additionally, 
the  D,T,SE&E  commits  OSD  to  provide 
appropriate  resources  to  complete  the  JT&E.  In 
turning  to  the  Services  for  the  required 
personnel,  the  FSD  and,  in  turn,  the  JTD  must 
recognize  that  the  Services'  ability  to  support  the 
requirement  will  be  largely  dependent  on  timing 
and  normal  rotation  cycles.  Coordination  by  the 
FSD  with  the  Service  personnel  support  POCs 


and  personnel  management  centers  during  the 
conduct  of  the  JFS  can  alleviate  some  of  these 
problems.  This  early  coordination  enables  the 
Service  personnel  centers  to  plan  and  prepare  for 
the  potential  requirements  of  the  chartered  JTF. 
This  preparation  would  include  the  identification 
of  personnel  who  would  be  immediately  available 
and  perhaps  the  alerting  of  personnel  to  the 
potential  assignment  so  that  they  are  prepared  to 
relocate  on  short  notice. 

The  JFS  must  take  a  hard  look  at  the 
Service  skills  and  experience  required  for 
conduct  of  the  JT&E.  Experience  in  past  JTFs 
would  indicate  that  the  FSD  should  recommend 
that  all  personnel  be  assigned  full-time  on  a  PCS 
basis  and  that  all  should  be  stationed  at  the  JTF 
facility.  The  JFS  must  recognize  that,  even  with 
early  coordination  with  the  Service  personnel 
centers,  acquiring  military  personnel  on 
permanent  assignment  to  the  JTF  may  involve  a 
period  of  several  months  before  the  person 
actually  arrives  for  duty.  In  such  cases,  and 
particularly  if  the  required  skills  and  experience 
are  critical  during  the  initial  phases  of  the  JT&E, 
the  JFS  should  include  in  the  coordination 
process  the  requirement  for  personnel  with  the 
requisite  skills  on  a  temporary  basis  until  the 
permanent  personnel  arrive.  The  JFS  should 
strongly  recommend  that  all  JFS  personnel 
remain  in  place  and  transition  to  the  JTF. 
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CHAPTER  5 

CLOSE  DOWN/TRANSITION  TO  JTF 


A.  INTRODUCTION 

Close  down  is  the  last  phase  of  the  JFS  and 
perhaps  the  most  demanding  in  terms  of  the  work 
to  be  accomplished  within  the  time  available. 
The  FSD  should  initiate  the  development  of  a 
simple  Close  Down  Plan  (CDP)  early  in  the  JFS 
to  identify  those  actions  that  must  be 
accomplished  and  a  schedule  for  their 
accomplishment  in  a  logical  order.  The  identified 
actions  will  include  the  preparation  and 
coordination  of  the  final  report,  transition  or 
release  of  personnel,  preparation  of  efficiency 
reports,  awards  and  recognitions,  the  transfer  of 
property,  termination  of  contracts  and  support 
agreements,  the  close  out  of  fiscal  accounts,  and 
the  return  or  transfer  of  facilities. 

B  .  FINAL  REPORT  COORDINA¬ 
TION  AND  BRIEFINGS 

Preparation  of  the  JFS  final  report  should 
start  early  and  continue  in  parallel  with 
completion  of  JFS  activities.  Some  of  the 
report  is  boilerplate  that  can  and  should 
be  developed  as  early  as  possible. 
Sections  of  the  report  should  be  completed  as 
data  and  information  are  collected  and  analyzed, 
and  conclusions  are  reached  relative  to  the 
feasibility  issues.  A  final  draft  of  the  report 
should  be  completed  in  sufficient  time  to  allow 
for  coordination  with  OSD,  Joint  Staff,  the  SAC, 
and  the  Services.  This  coordination  should  be  in 
accordance  with  the  coordination  chain 
previously  established.  It  should  be  noted  that  it 
has  not  been  uncommon  for  this  coordination  to 


take  up  to  six  weeks  with  the  resulting  revisions 
and  follow-on  coordination  taking  another  two 
weeks. 

The  FSD  should  present  briefings  on  the 
conclusions  and  recommendations  of  the  JFS  to 
key  DoD,  Joint  Staff,  and  Service  organizations 
in  conjunction  with  coordination  of  the  JFS  final 
report  and  APA.  These  briefings  are 
particularly  important  for  gaining 
acceptance  and  support  of  the  JFS 
recommendations.  Conduct  of  these  briefings 
can  be  time  consuming  and  will  necessitate 
planning  and  coordination  to  maximize  the 
number  of  briefings  that  can  be  accomplished  on 
each  trip.  Read-ahead  copies  of  the  briefings 
should  be  provided  to  the  respective 
organizations  in  order  to  maximize  the  available 
time. 

The  final  report  and  the  APA  will  be 
presented  to  the  TAB  for  concurrence  and  to  the 
DDT&E  for  approval. 

Finalization,  production,  and  distribution  of 
the  approved  JFS  final  report  should  be 
completed  at  least  two  weeks  prior  to  the 
scheduled  close  down  date.  Any  continuation  of 
work  on  these  documents  after  that  date  can  be 
affected  by  the  administrative  and  logistical 
actions  associated  with  the  close  down.  These 
actions  include  the  transfer  of  the  remaining 
assets  (computers,  printers,  ADP,  desks  and 
office  equipment,  etc.)  back  to  the  controlling 
organizations  or  agencies  and  the  transfer  of  the 
facilities  back  to  the  host  installation. 
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C .  PERSONNEL 

Most  of  the  JFS  personnel  will  have  been 
assigned  on  a  TDY  basis.  If  the  JFS  results  in  a 
chartered  JTF,  the  JTF  will  require  full-time 
personnel,  and  the  Service  personnel  systems  can 
take  up  to  14  months  to  provide  permanent 
change  of  station  (PCS)  personnel.  The  JTF 
may  have  to  operate  for  roughly  one-half  of  its 
life  cycle  before  full  staffing  of  Service  personnel 
occurs.  Initial  JTF  activities  will  depend 
primarily  on  the  expertise  and  resources  that  can 
be  transitioned  from  the  JFS  and  what  can  be 
obtained  through  contractor  support.  The  FSD 
must  consider  this  in  the  development  of  his 
close  down  plan.  The  FSD  should  recommend 
that  whenever  possible,  JFS  personnel  and 
resources  should  transition  to  and  become  the 
core  element  of  the  JTF.  The  JFS  should  also 
note  and  comment  on  the  experience  and 
expertise  developed  by  support  contractors 
during  the  JFS. 

In  the  event  that  JTF  charter  probabilities 
are  low,  the  FSD  should  develop  a  schedule  for 
the  release  of  all  assigned  personnel.  Service 
personnel  will  return  to  their  parent 
organizations,  and  Civil  Service  personnel  will 
be  assigned  to  organizations  on  the  host 
installation  in  accordance  with  this  schedule.  The 
FSD  should  alert  the  host  installation  Human 
Resource  Office  (HRO)/Human  Resources 
Department  (HRD)/Civilian  Personnel  Office 
(CPO)  of  the  projected  JFS  completion  date  as 
early  as  practical  and  of  the  necessity  to  find  jobs 
for  the  Civil  Service  personnel.  Contractor 
personnel  are  a  unique  asset  and  often  constitute 
the  most  stable  personnel  resource  available  to 
the  JFS  during  close  down  because  they  do  not 


have  the  same  reassignment  pressures  as  the 
military  or  Civil  Service  personnel.  Thus,  they 
will  most  likely  remain  available  to  support  the 
JFS  up  to  the  closing  date.  Contractor  personnel 
may  also  be  required  to  fill  voids  created  by  the 
early  release  of  military  personnel.  The  use  of 
contractor  personnel  to  provide  this  augmentation 
may  necessitate  a  revision  of  the  support  subtask 
to  assure  that  the  required  support  is  within  the 
scope  of  the  contract. 

During  the  close  down  phase,  the  FSD 
should  give  special  attention  to  personnel 
efficiency  and  appraisal  reports.  In  that  the 
personnel  are  assigned  on  a  TDY  basis,  this 
involves  providing  inputs  to  the  individual's 
rating  official.  In  the  case  of  both  military  and 
Civil  Service  personnel,  failure  to  provide  timely 
input  to  their  appraisal  or  efficiency  report  could 
result  in  the  completion  of  an  annual  appraisal  by 
a  supervisor  who  has  had  little  or  no  time  to 
observe  the  individual's  performance.  The 
intensity  of  activity  during  the  close  down  phase 
and  the  focus  on  work  completion  often  result  in 
appraisal  inputs  being  overlooked  or  hastily 
provided.  Once  again,  the  CDP  should  include 
time  for  preparation  of  quality  inputs  into  the 
military  and  Civil  Service  personnel  appraisal 
systems. 

As  the  JFS  nears  completion,  attention 
needs  to  be  given  to  the  recognition  of  those 
personnel  and  organizations  that  have  provided 
extraordinary  support  to  the  JFS.  Since  the 
military  personnel  will  be  in  a  TDY  status,  award 
recommendations  or  recognitions  will  most  likely 
be  forwarded  to  the  parent  units.  Civil  Service 
recognitions  should  be  coordinated  with  the  host 
installation  HRO/HRD/CPO. 
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D.  DISPOSITION  OF  ASSETS 

Close  down  of  the  JFS  will  involve  the 
disposition  of  property  and  facilities.  If  the 
charter  of  a  JTF  is  probable,  the  FSD  should 
make  every  effort  to  transfer  all  JFS  property  and 
products  to  the  JTF.  This  includes  all 
administrative  and  technical  library  documents, 
databases,  forms,  procedures,  etc.,  that  were 
acquired  or  developed  during  the  JFS.  If  charter 
is  not  probable,  the  FSD  should  plan  to  dispose 
of  these  assets  in  accordance  with  the  appropriate 
policies  and  procedures.  Contact  the  OSD  JT&E 
Coordinator  for  disposition  information  and 
instructions  related  to  JFS-unique  material. 

Property 

The  JFS  may  end  up  with  two  different 
owners  of  property  that  must  be  disposed  of, 
Service  and  OSD. 

Service  Property  provided  or  purchased 
by  a  Service  remains  the  property  of  that  Service 
and  will  be  returned  when  no  longer  needed. 
Return  of  the  property  should  be  coordinated 
through  the  Service  JT&E  POC  who  will  define 
disposition  procedures.  An  inventory  of  all  such 
property  will  be  conducted  prior  to  the  physical 
transfer. 

OSD  Property  provided  by  OSD  or 
purchased  with  DDT&E  funds  is  the  property  of 
DDT&E.  OSD  property  will  be  returned  to  OSD 
control  when  no  longer  required  for  the  JFS 
mission.  An  inventory  of  all  such  property  will 
be  conducted  prior  to  the  transfer.  In  the  case  that 
a  follow-on  JTF  is  probable,  the  FSD  will 


coordinate  with  the  OSD  JT&E  Coordinator 
relative  to  the  transfer  of  JFS  property  to  the 
JTF.  In  no  case  should  the  FSD  take  any  action 
or  make  any  commitments  regarding  the 
disposition  of  OSD  property  to  another  JFS, 
JTF,  or  other  organization  without  prior 
coordination  and  approval  of  DDT&E. 

Facilities 

The  return  of  JFS  facilities  will  be  in 
accordance  with  procedures  coordinated  between 
the  FSD,  the  lead  Service  JT&E  POC,  and  the 
host  base  installation  office. 

E.  JFS  PRODUCTS 

The  JFS  will  produce  products  that  have 
value  to  areas  other  than  the  chartering  decision 
process.  Such  products  include  models, 
simulations,  and  databases  used  or  produced 
during  the  JFS. 

Models  and  Simulations 

If  the  JFS  has  adapted  or  developed  a 
model  for  its  use,  it  represents  a  product  of  the 
JFS.  The  lead  Service  will  retain  a  copy  of  all 
models  and  simulations  used  in  JT&E  programs 
for  use  in  future  JT&Es  that  may  have  a  need  for 
them.  A  JFS  that  acquires,  modifies,  or 
develops  a  model  or  simulation  will  provide  the 
lead  Service  modeling  and  simulation  agent  with 
the  model  or  simulation,  to  include  the 
documentation  (user  instructions  and  technical 
details  regarding  the  model's  structure)  required 
for  subsequent  use  or  modification  of  the  model 
or  simulation.  The  FSD  will  provide  the  lead 
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Service  modeling  and  simulation  agent  with  the 
identification  of  those  agencies  that  have 
expressed  an  interest  in  the  model  and  a  list  of 
agencies  to  which  copies  of  the  model  have 
already  been  provided. 

Data  and  Databases 

Considerable  JFS  efforts  were  expended  in 
the  collection  of  data  and  development  of  a 
database.  These  data  represent  a  product  that 
should  be  archived  and  protected  from  corruption 
to  ensure  their  availability  to  other  JT&E  and 
Service  programs.  If  a  JT&E  charter  is  probable, 
all  JFS  data  and  databases  should  be  transferred 
to  the  JTF.  As  with  models  and  simulations, 
each  Service  has  a  principal  recipient  of  JT&E 
databases  and  controls  their  transfer  to  other 
agencies.  The  FSD  will  also  identify  agencies 
that  have  expressed  an  interest  in  the  JFS 
databases  (or  portions  thereof)  and  agencies  to 
which  copies  have  already  been  provided. 

Documents  and  Reports 

All  technical  documents  (observations, 
findings,  recommendations,  results)  and  reports 
produced  by  the  JFS  will  be  approved  by 
DDT&E  prior  to  archiving  in  supporting  technical 
libraries.  Copies  of  all  approved  technical 
documents  and  reports  should  be  forwarded  to 
the  AFOTEC  library  for  permanent  storage.  The 
address  for  the  AFOTEC  library  is  listed  in 
Chapter  2.  Copies  of  all  DDT&E  approved 
technical  documents  and  reports  that  are  pertinent 
to  the  DoD  mission,  contribute  to  the  DoD  or 


national  scientific  or  technological  base  should 
also  be  forwarded  to  the  Defense  Technical 
Information  Center  (DTIC).  Procedures  for 
submission  of  technical  documents  and  reports  to 
DTIC  can  be  obtained  by  contacting: 


Defense  Technical  Information  Center 
(DT1C70CP) 

Cameron  Station 
Alexandria,  VA  2304-6145 
Telephone:  (703)  224-6847 


F.  CONTRACTS  AND  SUPPORT 

AGREEMENTS 

All  contracts  and  support  agreements  that 
were  established  by  the  JFS  should  be  closed 
out.  Some  may  require  special  actions  associated 
with  the  close  out.  The  OSD  JT&E  Coordinator 
is  skilled  in  contract  negotiations  and  is  available 
to  assist  in  this  area.  One  of  the  actions  required 
in  contract  close  out  is  a  review  of  the  contract 
relative  to  completion  date,  level  of  effort,  and 
scope.  The  contract  should  be  evaluated  in  terms 
of  JFS  requirements  yet  to  be  accomplished, 
remaining  resources  and  level  of  effort  to 
accomplish  these  tasks,  and  the  funding  available 
to  complete  the  work.  If  it  is  determined  that  no 
additional  support  is  required,  then  action  should 
be  initiated  to  close  out  the  contract(s).  The  FSD 
should  also  take  the  necessary  actions  to 
terminate  all  support  agreements  with  host- 
installation  and  supporting  organizations  and 
agencies. 


5-4 


JFS  Handbook 
February  1,  1996 


G .  CLOSE  OUT  OF  FISCAL 
ACCOUNTS 

Even  after  a  JFS  closes  its  doors,  the 
cleanup  of  financial  matters  will  continue.  Late 
billings  and  final  contract  dispositions  create 
requirements  for  funds  after  the  JFS  is 


disestablished.  The  OSD  JT&E  Coordinator  will 
provide  fiscal  assistance  relative  to  close  out 
activities  if  the  chartering  of  a  JTF  is  not 
expected.  If  chartering  is  expected,  the  cleanup 
of  JFS  bills  and  funding  should  transfer  to  the 
JTF. 
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ANNEX  A 

LIST  OF  ACRONYMS 


ACC 

Air  Combat  Command  (Air  Force) 

ACID 

Advanced  Concept  Technology  Demonstation 

ADP 

Automated  Data  Processing 

AFOTEC 

Air  Force  Operational  Test  and  Evaluation  Center  (Air  Force) 

ALSA 

Air  Land  Sea  Agency 

APA 

Analysis  Plan  for  Assessment 

CATEX 

Categorical  Exclusion 

CDP 

Close  Down  Plan 

CINC 

Commander  in  Chief 

COI 

Critical  Operational  Issue 

COMSEC 

Communications  Security 

CMC 

Commandant  of  the  Marine  Corps  (Marine  Corps) 

CNO 

Chief  Naval  Operations  (Navy) 

CPO 

Civilian  Personnel  Office 

CPX 

Command  Post  Exercise 

CRE 

Consolidated  Resource  Estimate 

DESA 

Defense  Evaluation  Support  Activity 

DCSOPS 

Deputy  Chief  of  Staff  for  Operations  (Army) 

DC/S  R&P 

Deputy  Chief  of  Staff  for  Requirements  and  Programs  (Marine  Corps) 

DDT&E 

Deputy  Director,  Test  and  Evaluation 

DIA 

Defense  Intelligence  Agency 

DMAP 

Data  Management  and  Analysis  Plan 

DoD 

Department  of  Defense 

DoDD 

Department  of  Defense  Directive 

D,T,SE&E 

Director,  Test,  Systems  Engineering  and  Evaluation 

DTD 

Deputy  Test  Director 

EA 

Environmental  Assessment 

EIS 

Environmental  Impact  Statement 

EPA 

Environmental  Protection  Agency 

FDMC 

Fiscal  Director  Marine  Corps  (Marine  Corps) 

FFRDC 

Federally  Funded  Research  and  Development  Center 

FMF 

Fleet  Marine  Force  (Marine  Corps) 

FOA 

Field  Operating  Agency  (Air  Force  and  Army) 

FONSI 

Finding  of  No  Significant  Impact 

FSD 

Feasibility  Study  Director 

FY 

Fiscal  Year 

FYTP 

Five  Year  Test  Plan 

GOSC 

General  Officer  Steering  Committee 

HQMC 

Headquarters  Marine  Corps  (Marine  Corps) 

HRD 

Human  Resource  Department 

HRO 

Human  Resource  Office 

IDRL 

Integrated  Data  Requirements  List 

IPR 

In-Process  Review 

JADO/JEZ 

Joint  Air  Defense  Operations/Joint  Engagement  Zone 

JCCD 

Joint  Camouflage  Concealment  &  Deception 

JCS 

Joint  Chiefs  of  Staff 
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JDC 

JFS 

JMEM 

JPO 

JT 

JT&E 

JTAMS 

JTD 

JTF 

MACOM 

MAJCOM 

MCO 

MCOTEA 

MCSYSCOM 

MCSYSCOM/T&E 

MLM 

MOA 

MOE 

MOT&E 

MOP 

NEPA 

N091 

N912D2 

O&M 

OPSEC 

OPTEC 

OPTEVFOR 

OSD 

OTA 

OT&E 

OTP 

PC 

PCS 

PID 

POC 

POM 

PTP 

PY 

QC 

SAC 

SCG 

SCI 

SCIF 

SME 

sso 

T&E 

TAB 

TAG 

TDY 

TEIN 

TRP 

TSARC 

WBS 


Joint  Doctrine  Center 

Joint  Feasibility  Study 

Joint  Munitions  Effectiveness  Manual 

Joint  Program  Office 

Joint  Test 

Joint  Test  and  Evaluation 
Joint  Tactical  Missile  Signature 
Joint  Test  Director 
Joint  Test  Force 
Major  Command  (Army) 

Major  Command  (Air  Force) 

Marine  Corps  Order 

Marine  Corps  Test  and  Evaluation  Agency  (Marine  Corps) 

Marine  Corps  Systems  Command  (Marine  Corps) 

Marine  Corps  Systems  Command/Test  and  Evaluation 

Mission  Level  Measure 

Memorandum  of  Agreement 

Measure  of  Effectiveness 

Multi-service  Operational  Test  and  Evaluation 

Measure  of  Performance 

National  Environmental  Policy  Act 

Office  of  Director  of  Test  and  Evaluation  (Navy) 

Office  of  Director  of  Test  and  Evaluation  and  Technology  Requirements  Test 
and  Evaluation  Division  (Navy) 

Operations  and  Maintenance 
Operational  Security 

Operational  Test  and  Evaluation  Command  (Army) 

Operational  Test  and  Evaluation  Force  (Navy) 

Office  of  Secretary  of  Defense 
Operational  Test  Agencies 
Operational  Test  and  Evaluation 
Outline  Test  Plan  (Army) 

Planning  Committee 
Permanent  Change  of  Station 
Program  Identification  Document 
Point  of  Contact 

Program  Objective  Memorandum 

Program  Test  Plan 

Program  Year 

Quality  Control 

Senior  Advisory  Council 

Security  Classification  Guide 

Special  Compartmented  Information 

Special  Compartmented  Information  Facility 

Subject  Matter  Expert 

Special  Security  Office 

Test  and  Evaluation 

Technical  Advisory  Board 

Technical  Advisory  Group 

Temporary  Duty 

Test  and  Evaluation  Identification  Number  (Navy) 

Test  Resources  Plan  (Air  Force) 

Test  Schedule  and  Review  Committee  (Army) 

Work  Breakdown  Structure 
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ANNEX  B 

GLOSSARY  OF  TERMS 


Analysis  Plan  for  Assessment  (APA) 

A  document  published  by  the  JFS  team  that 
stipulates  what  actions  must  be  accomplished  by 
a  JTF  in  order  to  collect  and  analyze  sufficient 
data  to  resolve  the  JT&E  issues  and/or  concepts. 
The  APA  specifies  what  must  be  done  in 
sufficient  detail  so  that  the  JFS  team  can  quantify 
the  resources  required  to  accomplish  the  JT&E. 

Battalion  Training  Strategy  (OPTEMPO) 

The  actual  cost  of  parts  and  fuel  to  operate 
annual  miles  for  vehicles  and  hours  for  aircraft 
executing  training  in  the  field  army. 

Critical  Operational  Issues  (COI) 

The  major  issues  (questions/problems)  that 
a  JT&E  is  addressing,  usually  phrased  in  the 
form  of  a  question. 

Data  Management  and  Analysis  Plan 
(DMAP) 

A  program  level  document  that  details  what 
data  will  be  collected,  how  it  will  be  cataloged, 
stored,  controlled,  analyzed,  and  archived.  The 
program  DMAP  also  contains  detailed  procedures 
for  these  functions.  A  DMAP  may  also  be 
required  to  detail  what  and  how  data  will  be 
collected,  managed,  and  analyzed  for  a  specific 
test  activity. 

Defense  Evaluation  Support  Activity 
(DESA) 

A  DoD  support  organization  available  to 
JFS  and  JTF  teams  for  advice  and  support 
regarding  virtually  all  aspects  of  JT&E. 


Designated  Support  Agent 

That  organization  or  Agency  that  is 
designated  by  competent  authority  to  provide 
support  to  a  JFS.  Frequently  referred  to  as  the 
support  agent. 

Environmental  Assessment  (EA) 

A  study,  required  by  the  National 
Environmental  Policy  Act  (NEPA),  to  determine 
if  significant  environmental  impacts  are  expected 
from  a  proposed  activity. 

Environmental  Impact  Statement  (EIS) 

A  report  required  by  NEPA  that  describes 
the  environmental  consequences  of  a  proposed 
activity. 

Feasibility  Study  Team 

The  personnel  who  perform  a  JFS. 

Feasibility  Study  Director  (FSD) 

The  person  appointed  as  responsible  for  the 
conduct  of  the  JFS. 

Free  Play 

Activities  of  a  player  staff  as  it  responds  to 
stimuli  provided  by  the  control  structure  of  a 
“game,"  where  the  stimuli  are  logical 
consequences  of  previous  player  actions.  In  this 
context,  a  game  is  some  representation  of  the  real 
world  intended  as  context  for  training  of  the 
player  staff. 

General  Officer  Steering  Committee 
(GOSC) 

A  group  of  General  Officers  from  the 
Services  interested  in  a  particular  JT&E  issue  or 
operational  concept  who  are  invited  by  the  FSD 
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to  advise  on  issues  of  doctrine,  policy,  or  tactics. 
The  intent  is  to  capture  and  integrate  Service 
representation  in  the  JFS  at  a  senior  officer  level. 

Instrumentation 

Equipment  used  during  a  test  activity  to 
capture  and  record  data. 

Issue 

A  question  that  a  nomination  poses  that  a 
JT&E  will  resolve.  The  measures  and  data 
elements  are  designed  to  resolve  the  issues. 

Joint  Feasibility  Study  (JFS) 

A  formal  study  undertaken  to  determine 
whether  a  proposed  JT&E  should  be  chartered 
for  execution  by  a  Joint  Test  Force  (JTF). 

Joint  Test  Director  (JTD) 

The  person  appointed  as  responsible  for 
executing  a  charted  JT&E  and  directing  the 
efforts  of  the  JTF.  Should  be  at  least  an  0-6 
(possibly  GS-15)  with  test  and  evaluation 
experience  and  a  background  in  the  technical 
aspects  of  the  nomination. 

Joint  Test  and  Evaluation  (JT&E) 

Those  activities  dedicated  to  addressing  an 
issue  or  concept  that  was  nominated  by  OSD, 
Joint  Staff,  CINC,  or  the  Services  and  has  been 
chartered  by  OSD  to  be  conducted  by  a  JTF. 

Joint  Test  and  Evaluation  (JT&E) 
Program 

Those  activities  managed  by  DDA&SP  for 
the  DoD,  including  all  active  and  proposed  JFSs 
and  JTFs.  Frequently  the  JT&E  program  is 
confused  with  the  actions  of  a  JTF,  which  are  to 
conduct  a  joint  test  and  evaluation  dealing  with  an 
issue  or  concept.  The  JT&E  program  includes  all 
functions  that  support  the  JT&E,  for  example  the 
convening  and  support  of  JT&E  planning 


committees,  SACs,  TABs,  GOSCs,  and  program 
and  budget  functions. 

Joint  Test  Force 

A  formal  organization  lead  by  a  JTD  and 
staffed  by  the  Services  for  a  specific  time  period 
to  conduct  a  JT&E  under  the  auspices  of  an  OSD 
charter. 

Measure  of  Effectiveness  (MOE) 

A  quantifiable  entity  that  expresses  the 
effectiveness  of  a  system  or  concept  under  test. 
An  MOE  can  also  be  defined  as  an  algorithm 
which  uses  data  to  be  collected  to  compute  a 
quantity  called  the  measure. 

Mission  Level  Measure  (MLM) 

A  quantitative  or  qualitative  measure  of  a 
systems  capabilities  or  characteristics  in  terms  of 
their  effect  on  the  mission  of  which  the  system  is 
a  part. 

Measure  of  Performance  (MOP) 

A  quantitative  or  qualitative  measure  of  a 
system's  capabilities  or  characteristics. 

Nomination 

The  process  used  to  bring  joint  issues  and 
concepts  to  the  attention  of  the  DDA&SP. 

Objectives 

Objectives  are  designed  to  focus  attention 
on  those  areas  that  require  or  involve  test 
activities  to  resolve  the  issues  or  concepts  being 
tested.  Activity  progress  can  be  measured  in 
terms  of  attaining  the  test  objectives. 

Outline  Test  Plan  (OTP)/Test  Resource 
Plan  (TRP) 

Resource  requirements  documents  used  by 
the  Army  and  Air  Force  respectively  for  users  to 
specify  personnel  and  equipment  requirements  to 
be  used  in  the  support  of  joint  test  activities. 
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Piggyback 

A  form  of  testing  where  a  joint  test  force 
uses  deployed  resources  of  one  or  more  Services 
for  testing  before,  during,  or  after  a  concurrently 
scheduled  Service  or  Joint  Staff  training  exercise 
in  order  to  collect  data. 

Program  Analysis  Methodology 

That  portion  of  the  PTD  that  defines  what 
analysis  must  be  done  using  the  collected  data 
elements  to  address  all  of  the  issues. 

Program  Test  Plan  (PTP) 

The  document  that  specifies  in  detail  the 
procedures  for  conducting  the  test  activity; 
collecting,  processing,  verifying,  storing,  and 
transporting  the  required  data;  and  producing  the 
required  reports. 

Quick-Look 

Those  procedures  established  to  assure  that 
the  amount  and  quality  of  data  being  collected 
during  test  activities  is  adequate. 

Reconstruction 

A  posttest  analysis  process  used  to  verify 
the  accuracy  of  collected  data  by  alignment  of  the 
data  to  test  activities. 

Senior  Advisory  Council  (SAC) 

An  advisory  body  that  reviews  selected 
nominations,  the  results  of  JFSs  and  JTFs,  and 
recommends  appropriate  actions  to  the 
D,T,SE&E. 

Service  Deputy 

A  senior  person  appointed  by  a  Service  to 
participate  in  a  JFS.  Serves  as  a  functional 
member  of  the  JFS  while  representing  the  interest 


of  the  appointing  Service.  Should  be  an  0-5  or 
0-6  with  test  and  evaluation  experience  and  a 
background  in  the  technical  aspects  of  the 
nomination. 

Technical  Advisor 

A  JFS  member  designated  by  the  FSD  to 
advise  on  technical  matters  and  to  resolve  any 
technical  differences  of  opinion  within  the  JFS. 
The  technical  advisor  is  responsible  for  keeping 
JFS  activities  focused  on  chartered  concepts 
and/or  issues. 

Technical  Advisory  Board  (TAB) 

A  group  of  senior  scientists,  engineers,  and 
analysts  who  advise  the  DDA&SP,  SAC,  JT&E 
PC,  FSDs,  and  JTDs  on  technical  matters 
relevant  to  JT&Es. 

Technical  Advisory  Group  (TAG) 

A  advisory  body  formed  to  provide  direct 
technical  support  and  advice  to  a  JFS.  The  TAG 
composition  is  similar  to  the  TAB  but  is  not  as 
senior. 

Test  Activity 

A  collective  term  used  to  describe  a  total 
series  of  related  tests  or  studies  conducted  to 
collect  and  analyze  data.  Test  activities  can  range 
from  analysis  using  studies  and  models  to  field 
tests  that  involve  deployed  combat  units. 

Test  Manager 

A  JTF  member  responsible  for  the 
planning,  execution,  and  reporting  of  a  specific 
test  activity. 
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ANNEX  C 

POCs  for  JT&E  SUPPORT 


OSD 


JCS 


USAF 


USA 


USN 


USMC 


DESA 


DDT&E 

Tel  -  (703)  697-3406 

FAX  -  (703)  614-7040 

E-Mail  LEDESMAR@ACQ.OSD.MIL 


Joint  Staff/J8 

Tel  -  (703)  694-9759 

FAX  -  (703)  694-6601 


USAF/TEP 

Tel  -  (703)  695-0900 

FAX  -  (703)  695-0803 

E-Mail  BRECHWAJ@TEP.HQ.AF.MIL 


DCSOPS  (DAMO-FDR) 

Tel  -  (703)  697-4044 
FAX  -  (703)  614-2675 

E-Mail  AGOSTA@PENTEMH8.ARMY.MIL 


JT&E  Coordinator 

Tel  -  (703)  578-6581 

Fax  -  (703)  578-6583 

E-Mail  BLOOMELE@ACQ.OSD.MIL 


AFOTEC/XRJ 

DSN  -  246-5339 

Tel  -  (505)  846-5339 

FAX  -  (505)  846-5214 

E-Mail  MAGEEW@P3.AFOTEC.AF.MIL 

OPTEC  CSTE-OPM 

Tel  - (703)  681-6518 

DSN  -  289-6518 

FAX  -  (703)  681-7584 

E-Mail  YOUNGA@OPTECI.ARMY.MIL 


OP  912D2 

Tel  -  (703)  697-1485/0181 
DSN  -  227-1485/0181 
FAX  -  (703)  697-1070 


MCSYSCOM  Code  (TE) 

Tel  -  (703)  784-5964 

FAX  -  (703)  784-3432 

E-Mail  MEADP@MQG-SMTP3.USMC.MIL 


DESA-BMC 

Tel  -  (505)  262-4529 

Fax-  (505)  260-2759 

E-Mail  HOLMESB@DESA.OSD.MIL 
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ANNEX  D 

RELATED  JT&E  DOCUMENTS 


DoD  Instruction  5000.2 

This  DoD  Instruction  provides  the  basic 
framework  for  the  Test  and  Evaluation  of 
programs  sponsored  or  supported  by  OSD. 

DoD  Manual  5000.3-M-4 

This  manual  provides  a  description  of  the 
joint  test  and  evaluation  nomination  and  selection 
process,  describes  the  organizational  framework 
within  each  Service  which  supports  the  program, 
and  identifies  principal  participants  and  their 
respective  responsibilities. 

DoD  Manual  7110.1-MI 

This  manual  outlines  the  procedures  for  the 
distribution  and  use  of  OSD  funds  in  support  of 
OSD  sponsored  or  supported  projects. 

DoD  Manual  5220.22-M 

This  manual  provides  detailed  information 
relative  to  the  Government  Information  Security 
Program  for  safeguarding  classified  information. 

DoD  Directive  3200. 11-D 

This  directive  is  a  summary  of  Major  Range 
and  Test  Facility  capabilities. 

DoD  Directive  3200.12 

This  directive  outlines  the  mission, 
responsibilities,  and  functions  of  the  Defense 
Technical  Information  Center  (DTIC).  Copies  of 
all  approved  JFS  generated  technical  documents 
and  reports  should  be  forwarded  to  DTIC. 

DoD  Directive  4120.14 

This  directive  establishes  guidelines  relative 
to  Environmental  Concerns  for  DoD  programs. 


DoD  Directive  5230.24 

This  directive  establishes  the  requirement 
for  all  managers  of  technical  programs  to  assign 
distribution  statements  to  technical  documents 
generated  within  their  program. 

DoD  Directive  6050.1 

This  directive  is  a  list  of  activities  that 
previously  have  been  found  to  have  no 
detrimental  effects  on  the  environment  and  do  not 
require  an  EA  or  EIS. 

DIA  Regulation  55-3 

This  regulation  is  a  guide  for  obtaining 
threat  information  support. 

DoD  Regulation  5200. 1-R 

This  regulation  establishes  requirements 
relative  to  the  government  information  security 
program. 

Memorandum  of  Agreement  on  Multi- 
Service  Test  and  Evaluation  and  Joint 
Test  and  Evaluation 

This  MOA  provides  the  basic  framework 
for  T&E  conducted  by  two  or  more  Operational 
Test  Agencies. 

Joint  Feasibility  Study  Handbook 

This  handbook  consolidates  OSD  guidance 
and  direction,  information,  references,  and 
procedures  on  conducting  a  Joint  Feasibility 
Study. 

Joint  Test  and  Evaluation  Handbook 

This  handbook  consolidates  OSD  guidance 
and  direction,  information,  references,  and 
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procedures  on  conducting  a  Joint  Test  and 
Evaluation. 

AFOTEC  Instruction  99-101 

This  instruction  provides  the  guidelines  and 
procedures  for  the  AFOTEC  conduct  of  OT&E 
on  Air  Force  systems.  The  instruction  contains 
guidance  relative  to  resource  planning  and 
funding  of  Joint  Test  and  Evaluations. 

AFOTEC  Pamphlet  55-8 

This  pamphlet  is  a  guide  for  resource 
managers  and  test  managers  on  procedures  for 
preparing  the  Test  Resource  Plan  (TRP). 

Army  Regulation  15-38 

This  regulation  provides  guidance  and 
establishes  procedures  governing  the  Test 
Schedule  and  Review  Committee  (TSARC). 


Test  Schedule  and  Review  Committee 
Handbook 

This  handbook  provides  TSARC  members 
and  the  user  test  community  with  a  concise 
document  that  combines  information  from  many 
sources  for  use  in  preparing  Outline  Test  Plans 
(OTPs)  for  inclusion  in  the  Department  of  Army 
Five  Year  Test  Program  (FYTP) 

Marine  Corps  Order  5000.11B 

This  order  establishes  guidance  for  the  Test 
and  Evaluation  of  systems  and  equipment  to  be 
employed  by  the  Marine  Corps. 

Military  Standard  1806 

This  standard  provides  procedures  for 
marking  scientific,  technical,  engineering, 
production,  and  logistics  technical  data,  to  denote 
the  extent  to  which  they  are  available  for 
secondary  distribution. 
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ANNEX  E 

STRUCTURED  APPROACH  FOR  DEFINING  JOINT  TEST 

MEASURES 


A.  INTRODUCTION 

Ensuring  that  the  necessary  and  sufficient 
set  of  measures  is  identified  requires  a 
comprehensive  development  of  the  issues  and 
subissues.  The  issues  presented  at  nomination 
can  lead  to  a  JT&E  that  is  too  broad  to  be 
executable  within  resources  constraints  or  too 
narrow  to  address  the  problem.  This  annex 
presents  an  approach  to  translating  the  issues  into 
measurable  entities. 

B  .  TYPES  OF  JT&E 

Understanding  the  problem  to  be  solved  is 
essential  to  developing  a  structured  approach. 
The  problem  should  fit  into  one  of  the  five 
areas  defined  for  a  JT&E  Program  in 
DoD  5000.3-M-4.  These  are  to: 

•  Develop  and  analyze  testing  methodologies. 

•  Evaluate  technical  and  operational  concepts. 

•  Provide  information  on  system  require¬ 
ments  and  improvements. 

•  Examine  systems  interoperability. 

•  Evaluate  technical  or  operational  perfor¬ 
mance  under  realistic  conditions  of 
interrelated/interacting  weapons  systems. 

From  the  test  developer’s  point  of  view, 
these  fall  into  two  types  of  JT&Es:  Methodology 
development  and  "evaluation"  tests. 

•  Methodology  development  includes  all 
aspects  of  development  and  validation  of 
the  methodology  concepts  and  procedures. 
Methodology  development  does  not  usually 


involve  systems  development,  but  systems 
integration  may  be  necessary,  and 
interfaces  and  assessment  tools  may  need  to 
be  developed. 

•  "Evaluation  tests"  involve  comparisons 
between  two  or  more  systems,  concepts,  or 
sets  of  procedures.  An  "evaluation"  test 
may  rate  systems,  concepts,  or  procedures 
against  a  set  of  criteria  and  may  require 
development  of  that  criteria. 

These  two  classes  of  JT&Es  involve 
different  analysis  methodologies  and 
consequently  different  types  of  structured 
approaches.  Further,  JT&Es  vary  widely  in 
scope  and  may  fall  into  one  category  or  the  other 
or  fall  along  a  continuum  involving  both 
methodology  development  and  "evaluation"  tests. 

A  good  example  of  a  nearly  pure 
methodology  development  JT&E  is  the  Joint 
Tactical  Missile  Signature  (JTAMS)  joint  test. 
JTAMS  was  chartered  to  develop  methods  and 
procedures  for  determining  missile  signatures, 
establish  a  database  of  missile  signatures,  and 
structure  a  method  for  coordinating  missile 
signature  requirements.  These  issues  all  relate  to 
building  a  capability  to  satisfy  a  stated  need  and, 
therefore,  require  a  problem  resolution  approach. 

An  example  of  a  nearly  pure  "evaluation" 
JT&E  is  the  Joint  Air  Defense  Operations/Joint 
Engagement  Zone  (JADO/JEZ)  joint  test. 
JADO/JEZ  was  chartered  to  determine  how  well 
a  JEZ  can  be  implemented  to  increase 
effectiveness  of  air  defense  forces  and  to 
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determine  under  what  conditions  a  JEZ  is  more 
effective  than  current  air  defense  doctrine,  tactics, 
and  procedures.  JADO/JEZ  represents  the  more 
classic  operational  evaluation  test.  The  purpose 
of  this  is  to  make  comparisons  between  different 
concepts  in  order  to  determine  which  is  more 
effective. 

Examples  of  structured  breakdowns  from 
previous  joint  feasibility  studies  and  joint  test  and 
evaluations  are  contained  in  the  PTDs  for  those 
JT&Es.  Copies  of  these  PTDs  can  be  obtained 
from  the  JT&E  library. 

The  point  is  that  different  types  of  JT&Es 
use  different  analysis  methodologies  and  this 
difference  extends  to  the  development  of  the 
structured  approach.  Each  potential  JT&E  will 
take  on  a  character  that  reflects  the  type  of  issue 
being  investigated.  An  issue  on  joint 
interoperability,  for  example,  requires 
development  of  measures  that  will  assess  how 
systems  or  activities  perform  or  support  one 
another.  An  issue  relating  to  the  development  of 
a  joint  testing  procedure  must  be  approached  in  a 
manner  that  defines  measures  that  lead  to  the 
establishment  of  a  comprehensive  test  method. 

Methodology  development  issues  can 
be  resolved  by  other  test  methods,  such  as 
development  of  software  and  hardware  or 
analysis  of  available  information.  The  approach 
for  methodology  development  issues  will  be  a 
sequential  step-by-step  plan  for  defining  and 
developing  the  procedures  and  methodologies 
needed  to  satisfy  the  requirement. 

" Evaluation "  test  issues  lend 
themselves  to  investigation  and  resolution  by 
identification  of  attributes  that  are  quantifiable 
and  measurable.  The  structured  approach  for  an 
"evaluation"  test  issue  establishes  a  data  trace  that 


ties  planned  collection  of  data  elements  to  the 
issues. 

The  process  of  systematically  breaking 
major  test  issues  into  smaller  segments  until  the 
attributes  of  each  segment  can  be  quantitatively 
answered  is  similar  for  all  JT&E  issues. 
However,  the  focus  for  a  methodology 
development  issue  is  on  defining  the  necessary 
steps  for  satisfying  the  step-by-step  procedures 
needed  to  produce  an  item  to  meet  requirements. 
The  focus  for  an  "evaluation"  test  issue  is  on 
gathering  data  to  make  comparisons.  Therefore, 
the  analytical  structure  takes  on  a  different 
character.  A  development  task  structure  is  one 
geared  to  ensuring  achievement  of  a  goal  and 
production  of  a  product.  The  "evaluation"  task 
structure  is  configured  with  steps  to  permit 
determination  of  relative  values. 

C.  TEST  STRUCTURE  DEVELOP¬ 
MENT 

Development  of  a  structured  approach  is  the 
first  step  in  the  plan  for  addressing  the  joint 
feasibility  study  goals,  objectives,  and  issues. 
Time  and  resources  will  most  likely  preclude  the 
structured  approach  from  addressing  every  aspect 
of  the  concept  or  problem.  The  approach  should, 
therefore,  include  sensitivity  analysis  and  trade¬ 
off  studies  of  the  tasks  in  the  JT&E  concept  to 
identify  and  quantify  those  specific  areas  that 
show  the  greatest  potential  for  resolution  of  the 
JT&E  issues(s).  This  structure  becomes  the 
basis  of  the  other  planning  tasks,  i.e.,  selection 
of  the  test  methods,  development  of  the  test 
method  matrix,  and  identification  of  resource 
needs.  Each  of  these  tasks  requires  a  thorough 
understanding  of  the  problems  that  led  to  the 
feasibility  study  issues.  Without  this 
understanding  of  the  environment,  it  is 
impossible  to  structure  an  approach  that 
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encompasses  all  the  aspects  that  must  be 
examined  in  adequately  addressing  the  issues. 

Whatever  the  type  of  the  test  issue,  a 
structured  approach  should  be  developed  to 
formalize  the  process  that  will  provide 
information  about  the  issue.  One  approach  to 
defining  how  issues  will  be  addressed  is  the 
establishment  of  a  hierarchy  in  which  the  issues 
are  at  the  highest  level.  These,  in  turn,  are 
supported  by  subissues,  sub-subissues,  and, 
finally,  measures.  It  is  difficult  to  standardize 
labels  for  the  different  levels  of  this  structure 
because  the  four  Services  use  different  labeling 
conventions.  However,  most  examples  from 
Department  of  Defense  (DoD)  publications  show 
a  structure  that  develops  issues  from  the  problem 
that  the  JT&E  is  to  solve,  decomposes  the  issues 
to  a  level  that  they  can  be  addressed  by 
measurable  quantities,  develops  those  measures, 
and  defines  supporting  data  elements.  The 
discipline  of  using  this  analytical  hierarchy 
(dendritic)  type  of  architecture  may  appear  to 
thwart  creativity,  but  the  structure  it  provides  to 
the  test  design  process  outweighs  any  possible 
loss  of  creativity. 

The  thrust  of  this  breakdown  is  to 
subdivide  the  test  issues  into  more  explicit 
questions  or  subelements  that  can  be  addressed 
by  data  gathered  by  test  events.  The  formation  of 
measurable  attributes  may  occur  at  different 
levels  depending  on  the  issue.  That  is,  one  issue 
may  have  a  subissue  that  leads  immediately  to 
quantifiable  attributes,  while  another  issue  may 
need  to  be  subdivided  to  the  sub-subissue  level 
before  measurable  attributes  are  evident. 
Figure  E-l  outlines  the  attributes  of  the 
structured  approach  levels. 

The  structured  approach  process  fosters 
discipline  and  accountability.  To  implement  this 
approach  the  analyst  breaks  the  issue  down  into 


smaller  and  smaller  segments  by  asking,  "what 
do  I  need  to  know  to  answer  this  question?" 
Each  level  of  the  structure  is  given  a  title,  i.e., 
subissue,  sub-subissue,  measure,  or  data 
element.  However,  there  are  no  set  number  of 
levels.  This  process  of  building  this  structured 
method  establishes  the  analytical  approach  that 
will  be  used  to  address  the  issues.  Also,  the 
analytic  structure  preserves  the  organization  and 
logic  of  why  specific  data  are  required  and, 
therefore,  serves  as  an  audit  trail.  The  decisions 
made  in  establishing  the  subelements  at  each  level 
focus  the  planning  in  a  specific  direction.  The 
joint  feasibility  study  staff  must  understand  and 
consider  how  these  decisions  will  impact  the 
JT&E,  as  it  will  require  the  collection  of  specific 
data  during  the  planned  test  activities  and  require 
that  the  analysis  be  performed  in  a  particular 
manner.  Each  decision  has  major  resource 
considerations. 

The  structured  approach  process  can  be 
misleading  because  it  seems  to  imply  "just 
continue  to  analyze  and  breakdown  the  problem 
into  measurable  parts."  In  fact,  construction  is 
often  inductive,  not  deductive,  and  requires 
creativity. 

Implied  in  the  nomination  objectives  are 
mission-related  objectives  that  are  to  be  achieved 
to  successfully  address  the  problem  that  the 
JT&E  is  to  solve.  The  issue  focuses  the 
analysis,  but  the  analysis  must  also  consider 
these  overarching  objectives.  For  example,  the 
issue  could  be  to  find  a  means  to  reduce  resource 
(input)  requirements,  but  the  implied  objective 
would  be  to  maintain  the  output  quality  or  to 
optimize  the  cost-benefit  relationship. 
Conversely  the  issue  could  be  to  improve  the 
output,  and  the  implied  considerations  would 
involve  the  impacts  on  resource  requirements  and 
the  processes  and  organizations  involved. 
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ISSUES 

•  Questions  to  be  resolved  and  starting  point  for  structure 

•  Statement  of  issue  normally  part  of  the  nomination  submission 

•  If  multiple  issues,  then  each  is  addressed  individually 

•  Not  a  quantifiable  level 

•  Supported  by  level  labeled  "subissue"  or  "measure" 

•  Subissues  may  not  equally  support  the  issue 

SUBISSUE 

•  Usually  forms  the  second  level  of  structure 

•  Addresses  major  subelements  of  issue  (normally  part  of  nomination  submission) 

•  Not  a  quantifiable  level 

•  Answered  from  information  obtained  at  lower  levels  of  dendritic  via  an 
interpretation 

•  Supported  by  level  labeled  "sub-subissues"  or  "measures" 

•  Sub-subissues  may  not  equally  support  answering  issue 

SUB-SUBISSUE 

•  Subelement  of  subissue  (usually  developed  by  feasibility  study  team) 

•  Not  a  quantifiable  level 

•  Answered  from  information  obtained  from  measures  via  interpretation/application 

•  Supported  by  measures  that  are  quantifiable 

•  Measures  may  not  equally  support  answering  the  sub-subissue 

MEASURES  (MLMs,  MOEs,  or  MOPs) 

•  May  be  developed  in: 

-  Conventional  manner  -  defines  function  or  process  as  attributes  in  quantifiable 
terms 

-  "Reverse  engineering"  -  derived  by  decomposing  products  into  critical 
components  or  processes 

•  Addresses  major  subelement  (attribute)  of  sub-subissue 

•  Highest  quantifiable  level 

•  Usually  a  set  number  of  data  elements  required  to  address 

•  Confidence/completion  is  a  function  of  the  number  of  data  sets  accepted  as  valid 
for  use  in  analysis 


Figure  E-1.  Attributes  of  Structured  Approach  Components 


Consequently,  the  analysis  required  to 
resolve  an  issue  is  more  complex  than  simply 
turning  a  crank  to  factor  the  issue  into  ever 
smaller  subissues.  It  usually  will  require 
consideration  of  the  problem  from  a  more  holistic 
or  upward  approach  in  addition  to  the  downward 
approach.  Regardless  of  the  analytical  approach, 
there  will  be  a  limit  to  the  test  resources,  and 
therefore,  once  the  context  of  the  problem  is 
defined,  the  challenge  is  to  scope  the  test  design 
down  to  efficiently  address  the  problem. 


With  respect  to  military  systems  there  are 
usually  well-defined  missions  and  system 
objectives,  and  the  trace  from  measurable  system 
requirements  to  mission  objectives  is  derived  by 
analysis.  Decomposition  of  missions  into 
functional  processes  and  finally  into  system 
attributes  is  a  common  analytical  approach 
(sometimes  referred  to  as  a  dendritic  due  to  the 
dendritic  or  tree-like  form  of  the  resulting  analytic 
structure).  Unfortunately,  this  approach  usually 
results  in  a  complex  analytical  structure  that 
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defines  many  more  compositional  elements  than  a 
reasonable  test  could  measure.  The  problem  can 
be  simplified  by  considering  only  the  critical 
decomposition  elements  of  the  mission  or 
objective  function.  Inherently,  this  means  that 
the  elements  of  the  problem  are  not  equally 
weighted  and  the  analyst  has  either  conducted 
sensitivity  analyses  in  early  trade-off  studies  or 
has  applied  judgment  to  select  (or  deselect)  the 
critical  factors  that  will  be  measured  and 
consequently  drive  the  evaluation. 

The  application  of  analytical  judgment  is 
not  a  flaw,  it  is  a  practical  necessity.  In  fact, 
judgment  and  even  creative  intuition  may  be  more 
useful  in  developing  the  analytical  structure  than 
a  rigorous  methodology  such  as  early  trade-off 
studies  with  attendant  sensitivity  analysis. 
Judgment  should  not  be  eschewed,  but  should  be 
recognized  as  fundamental  to  the  analytical 
process,  and  the  rationale  and  sensitivities  of 
analytical  judgments  should  be  consciously 
addressed. 

If  the  analytical  structure  is  based  on  a 
decomposition  of  one  or  more  issues  or 
objective  functions,  it  is  important  to  recognize 
that  many  subelements  may  have  a  common 
attribute;  i.e.,  the  subelements  will  not  be 
mutually  exclusive  factors.  For  example,  system 
speed  could  contribute  to  many  functional 
processes  and  consequently  could  contribute  to 
mission  effectiveness  in  many  ways.  Thus  one 
would  expect  the  measure  of  system  speed  to  be 
a  factor  in  many  subelements,  and  the  evaluation 
must  deal  with  how  to  aggregate  the  impact  of 
system  speed. 

Another  consideration  when  using 
functional  decomposition  as  an  analytical 
approach  is  that  a  dendritic  is  not  a  good  model 
for  system  attributes  that  have  interactions  or 
interdependencies.  For  example,  night  vision 


and  muzzle  flash  suppression  both  contribute 
independently  to  mission  effectiveness,  but  there 
is  also  likely  to  be  a  synergistic  relationship  that 
is  not  apparent  from  a  simple  dendritic. 
Conversely,  good  cockpit  lighting  and  good 
night  vision  goggles  could  be  a  dysfunctional 
combination. 

If  it  is  recognized  that  determining  the  effect 
of  interactions  and  interdependencies  is 
fundamental  to  the  purpose  of  the  test,  the  test 
design  could  be  driven  more  by  an  articulation  of 
the  variables  and  consideration  of  what  might  be 
the  critical  combinations  of  the  variables.  In  that 
case,  it  may  be  more  useful  to  consider  the 
variables  of  interest  and  methods  to  model  these 
relationships  than  to  decompose  the  objective 
functions  implied  by  the  issue  statement. 

A  rigorous  analytical  structure  will  not 
compensate  for  lack  of  common  sense.  The 
dendritic  is  a  necessary  part  of  the  evaluation,  but 
it  must  be  related  back  to  mission-related, 
operational  considerations  to  be  meaningful. 
Some  factors  to  consider  include: 

•  The  analytical  structure  should  factor  explicit 
or  implied  overarching  objectives  into 
elements  that  comprise  an  objective  function. 

•  Many  of  the  sub-elements  of  a  rigorous 
functional  decomposition  may  not  be  critical 
to  the  issue  at  hand. 

•  The  statement  of  the  issue  focuses  the  test 
and  evaluation  on  those  sub-elements  that 
are  critical  to  resolve  the  issue. 

•  Care  should  be  taken  to  ensure  that  critical 
elements  defined  by  the  issue  are  not  a 
suboptimized  statement  of  overarching 
objectives. 

•  The  test  design  should  be  based  on 
measuring  the  critical  elements  in 
combinations  that  will  identify  the 
interactions  and  interdependencies. 
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While  exploring  different  avenues  to  satisfy 
test  requirements,  a  JFS  team  must  not  only 
determine  what  is  feasible,  but  also  the  minimal 
questions  that  must  be  answered.  Therefore,  the 
team  needs  to  focus  its  efforts  on  scoping  the 
planned  program  to  a  limited  number  of 
subissues  that  support  the  program  issues.  A 
highly  successful  method  for  selecting  subissues 
starts  with  a  screening  process  in  which  all 
suggestions  are  considered.  Frequently,  several 
proposed  sub  issues  can  be  combined  into  one. 
The  result  should  be  the  minimum  mutually 
exclusive  set  of  critical  factors  or  questions  that 
fully  comprise  the  issue. 

Frequently,  because  of  the  complexity  of  a 
JT&E  or  an  issue,  it  is  necessary  to  subdivide 
subissues  into  sub-subissues.  This  should  only 
occur  when  the  subissue  itself  is  so  complex  that 
it  can  logically  be  decomposed  into  non- 
quantifiable  questions.  It  is  possible  for  one 
subissue  to  require  sub-subissues  and  another 
subissue  under  the  same  issue  to  not  require 
further  breakout.  Any  path  from  issue  to  data 
element  does  not  require  the  same  number  of 
steps  as  any  other  path. 

D .  MEASURE  DEVELOPMENT 

Eventually,  an  analysis  structure  will  be 
defined  that  will  answer  the  issue  by  evaluation 
of  specific  well-defined  measures.  The  measure 
is  the  quantifiable  level  and  should  be  objective. 
The  conventional  manner  of  developing  measures 
is  to  extract  measure  statements  from  the 
subissue  or  sub-subissue  attributes.  Well- 
designed  measures  should  meet  the  criteria  listed 
in  Figure  E-2. 

Another  technique  that  helps  the  joint 
feasibility  study  team  to  focus  the  data  collection 
toward  satisfying  the  issues  and  to  communicate 


very  precisely  with  one  another  is  to  relate 
system  or  concept  attributes  to  measures.  Those 
attributes  that  lend  themselves  to  being  addressed 
by  a  graphical  display  can  be  expressed  in  terms 
of  a  subjective  evaluation  or  an  equation  that 
models  the  relationship  even  though  cause  and 
effect  are  not  established.  Then  the  data  elements 
needed  to  satisfy  the  equation  can  be  defined. 

Another  technique  that  may  have  some 
application  is  "reverse  engineering."  This 
technique  involves  the  identification  of  what  data 
must  be  generated  to  determine  whether  a  test  is 
successful  or  not.  Reverse  engineering  places 
emphasis  on  what  type  of  products  are  desired  to 
support  issue  resolution.  The  reverse 
engineering  approach  is  based  on  being  able  to 
visualize  the  contribution  of  system  attributes  in 
the  form  of  a  product  or  effectiveness.  In  order 
to  have  the  capability  to  generate  analytical 
displays,  the  attributes  must  be  expressed  in 
terms  of  quantifiable  data  that  can  be  collected 
and  associated  to  provide  a  measure  of  some 
capability.  By  knowing  the  type  of  decision  that 
must  be  reached  to  evaluate  an  attribute,  one  can 
determine  the  measurable  test  data  and 
assessments  that  must  be  used  to  arrive  at  a 
value,  or  range  of  values  of  effectiveness.  The 
required  measurements  and  data  elements  that 
must  be  collected  can  then  be  defined. 

Both  the  breakout  of  measures  via  the 
structured  approach  process  and  the  "reverse 
engineering"  process  achieve  the  same  goal,  the 
identification  of  data  that  must  be  collected  to 
answer  the  sublevel  question.  The  difference  is  a 
subtle  one.  While  the  traditional  approach 
focuses  on  the  data  that  needs  to  be  collected  to 
answer  an  issue,  the  "reverse  engineering" 
approach  focuses  on  what  should  be  reported  as 
a  logical  outgrowth  of  data  collected. 
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1.  The  measure  must  be  relevant.  Each  measure  must  directly  relate  to  the  issues  and 
test  design  that  have  been  identified.  The  measure  should  not  be  overly  broad  and 
should  involve  only  items  that  affect  the  issues  of  the  nominated  tests. 

2.  The  set  of  measures  should  be  complete  or  exhaustive.  Any  input  variable  that 
affects  the  system's  ability  to  perform  its  mission  should  appear  as  some  sort  of  input 
to  a  measure  and  should  cause  change  in  the  value  of  the  measure  as  it  is  varied  (i.e. 
Its  sensitivity  is  assessed  relative  to  input  variables  over  a  realistic  range  of  values). 

3.  The  measure  should  be  precisely  defined,  so  that  there  is  no  possibility  of 
misunderstanding  what  is  meant  by  the  measure. 

4.  The  set  of  measures  should  be  mutually  exclusive,  if  possible.  If  this  is  not  the 

case,  one  input  may  weigh  several  measures  and  thus  be  counted  several  times,  so  as 
to  have  a  disproportionate  effect  on  test  results. 

5.  The  measure  should  be  expressed  in  terms  that  are  meaningful  to  both  testers 
and  decision  makers.  This  test  must  be  clear  and  precise  to  ensure  that  the  meaning 
is  not  in  doubt  after  passage  of  time  and  examination  by  new  personnel. 

6.  Inputs  that  make  up  the  measure  should  be  measurable  in  some  fashion, 
preferably  quantifiably.  If  qualitative  measurements  must  be  used,  they  should  be 
judged  against  some  standard  criteria. 


Figure  E-2.  Desired  Qualities  in  a  Measure 


In  all  cases  the  analytical  structure 
developed  to  ensure  a  systematic  approach  to  the 
solution  provides  the  structure  for  tying  data 
element  requirements  to  the  analytical  approach. 
This  device  is  termed  a  "data  trace"  and  provides 
the  basis  for  supporting  the  collection  of  specific 
data  elements.  The  data  trace  also  proves  useful 
in  identifying  data  sources  and  instrumentation/ 
collection  requirements.  Data  requirements  not 
directly  tied  to  a  level  in  the  structured  approach 
must  be  examined  to  determine  their  value.  The 
data  trace  can  aid  determination  of  resource 
requirements  and  assessments  of  the  impact  of 
the  inability  to  obtain  data  due  to  the  absence  of  a 
test  vehicle  or  instrumentation. 

In  the  development  of  a  structured 
approach,  the  statement  of  the  questions  or 
measures  at  each  level  must  provide  an 


achievable  step-by-step  development  with  check 
points  as  to  the  suitability  of  the  development. 
For  an  evaluation  task,  the  structured  approach 
level  questions  are  configured  to  compartment  the 
subject  into  segments  where  common 
measurements  are  possible  and  one-to-one 
comparisons  can  be  made.  Therefore,  the 
developer  of  each  structured  approach  needs  to 
recognize  that,  based  on  the  orientation  of  the 
stated  issue,  different  approaches  are  needed. 
After  the  structure  has  been  developed  through 
the  measure  level,  Figure  E-3  gives  a  checklist 
that  can  be  used  in  assessing  the  adequacy  of  the 
developed  approach. 
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Figure  E-3.  Structure  Checklist 
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ANNEX  F 
TEST  METHODS 


A.  INTRODUCTION 

This  annex  describes  the  various  test 
methods  that  a  JT&E  could  employ;  in  effect,  the 
tools  that  are  available  to  address  the  program 
issues.  The  list  of  methods  is  lengthy,  but  not 
exhaustive.  There  are  two  basic  tools  for 
accomplishing  the  JT&E  efforts.  These  are  1) 
problem  solving,  and  2)  testing  (e.g. 
assessments,  evaluations,  or  validations).  Some 
of  the  relative  costs  and  benefits  associated  with 
each  method  are  discussed  in  the  following 
paragraphs  and  are  summarized  in  Figure  F-l. 
The  advisability  of  using  one  or  more  of  the 
methods  will  depend  on  specific  test 
requirements  and  on  such  factors  as  availability, 
credibility,  cost,  or  acceptance  of  test  results  by 
the  Services,  fidelity  of  the  test  data,  test  site 
location,  and  available  support. 

The  JFS  will  make  an  initial  determination 
of  test  methods  as  part  of  the  APA  development. 
JTF  coordination  for  the  actual  use  of  the 
resources  to  conduct  the  test  activities  may 
determine  that  some  are  not  available  when 
needed  or  some  are  too  costly. 

B .  PROBLEM  SOLVING 

Most  JT&Es  will  have  at  least  one  issue 
that  calls  for  fixing  a  problem  or  finding  a  better 
way.  Thus,  the  classic  problem  solving  process 
is  a  key  player  in  almost  all  JFSs.  This  includes 
bringing  together  people  capable  of  solving  the 
problem,  brainstorming  for  solutions,  analyzing 
and  ranking  potential  solutions,  and  selecting  the 


best  solution.  The  magnitude  of  this  effort  can 
vary  widely  with  the  complexity  of  the  problem. 

One  of  the  first  steps  in  most  problem 
solving  activities  is  information  collection.  This 
can  be  as  simple  as  referencing  previous  test  and 
exercise  results.  Review  of  published 
documentation  is  the  most  straightforward  way 
of  collecting  data  and  there  are  many  excellent 
sources  available  to  the  JFS.  These  include 
industrial  papers  and  studies,  Federally  Funded 
Research  and  Development  Center  (FFRDC) 
studies  and  analyses,  and  Service  programs  in 
the  same  technical  field.  Operational  records  are 
another  good  source  of  information.  A  drawback 
to  using  operational  data  is  that  it  may  require 
manipulation  of  the  data  to  reveal  the  information 
of  interest  and  may  lack  the  fidelity  or  accuracy 
needed. 

New  or  supplemental  information  may  be 
acquired  through  the  use  of  surveys  or  expert 
opinion,  or  the  use  of  validated  models  and 
simulations.  This  can  increase  the  effectiveness 
of  operation  records  by  filling  in  those  areas 
where  the  operational  records  are  weak. 
Validated  modeling  and  simulations  can  also  be 
used  to  develop  new  information  by  looking  at 
existing  data  in  new  or  different  ways. 

When  research,  modeling,  or  simulations 
do  not  provide  the  data  needed,  laboratory 
measurements  may  be  required.  With  laboratory 
measurements,  the  resulting  data  are  not 
compared  (evaluated)  against  a  set  of  criteria. 
Rather,  the  data  are  simply  collected  and 
reported. 
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SELECTION 

X.  FACTORS 
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NOTE:  Values  are  subjective  estimates  where  1  represents  the  most 
desirable  condition 


Nominal  Value  of  Testing  Options 


Figure  F-1. 

C .  TESTING 

JT&Es  are  typically  associated  with 
activities  to  collect  and  analyze  data  to  assess, 
evaluate,  or  validate  a  concept  or  system. 
Frequently,  JT&E  team  members  who  acquired 
their  testing  experience  with  a  Service  test 
organization  tend  to  think  of  testing  only  in 
regards  to  new  Service  systems,  i.e.  hardware 
and  software.  In  reality,  this  focus  is  too 
narrow.  For  example,  the  interoperability  of  two 
or  more  systems  can  be  assessed,  the 
effectiveness  of  an  operational  procedure  can  be 
evaluated,  or  a  new  or  revised  process  can  be 
validated.  JFS  staff  members  must  think  in  the 
broadest  terms  of  testing  and  how  test  activities 


will  contribute  to  the  resolution  of  the  program 
issues. 

An  assessment  provides  an  understanding 
of  the  relative  magnitude  of  what  is  involved  and 
determines  the  size  and  importance  of  that  which 
is  being  examined  in  terms  of  a  whole  picture. 
An  evaluation  determines  the  relative  worth  or 
significance  of  that  which  is  being  evaluated 
(there  are  usually  criteria  which  are  used  to  aid  in 
the  evaluation  process).  A  validation  provides  a 
means  for  corroborating  or  supporting  something 
based  on  a  sound  basis.  Validation  creates  the 
foundation  that  ties  the  subject  being  validated  to 
an  authority  on  that  subject. 
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The  various  methods  for  obtaining  data 
elements  to  support  assessments,  evaluations,  or 
validations  are  discussed  next. 

Surveys 

The  term  surveys  is  used  to  cover  a  range 
of  methods  including  questionnaire  and  interview 
methods  to  collect  data  about  system 
performance.  Frequently,  surveys  involve  the 
use  of  subject  matter  experts  but  may  also  be 
used  to  elicit  information  from  typical  users  of  a 
system.  Surveys  are  useful  when  evaluation 
criteria  are  ill-defined,  such  as  when  gauging 
general  improvements  of  a  process  or  assessing 
the  extent  of  a  problem.  Surveys  are  also  useful 
when  the  feature  under  scrutiny  does  not  lend 
itself  to  objective  analysis,  such  as  human  factors 
concerns. 

The  cost  for  conducting  surveys  is  typically 
much  lower  than  assessment  or  evaluation 
methods.  One  factor  that  must  be  considered, 
however,  is  the  cost  of  access  to  the  survey 
participants.  The  fidelity  of  a  survey  is 
dependent  on  the  quality  and  quantity  of  the 
responses  from  the  survey  participants. 

Laboratory  Tests 

Laboratory  tests  are  generally  the  least 
expensive  and  least  complex  type  of  tests  to  plan 
and  execute.  They  are  generally  used  to  obtain 
engineering  characteristics  or  specifications 
relative  to  the  performance  of  specific  systems 
under  laboratory-controlled  conditions.  Such 
tests  can  range  from  the  use  of  bench  tests,  to 
anechoic  chambers,  to  antenna  measurement 
facilities,  and  to  system  stress  test  facilities. 


In  addition  to  being  perhaps  the  least 
expensive  and  simplest  test  to  plan  and  conduct, 
lab  tests  provide  the  means  to  exercise  the 
greatest  control  over  the  test  and  can  be 
conducted  at  whatever  pace  is  necessary  to 
confirm  and  validate  the  data  collected. 
Equipment  can  be  left  in  its  test  configuration 
until  the  data  are  validated,  and  if  the  test  needs  to 
be  rerun,  the  equipment  is  already  setup  and 
ready  for  repeat  testing.  Nearly  all  of  the  testing 
conditions  can  be  rigidly  controlled  to  preclude 
any  outside  interference  on  system  operations 
and  the  data  collection  and  management  process. 
Testing  can  be  conducted  anytime  based  on  the 
availability  and  operating  hours  of  the  test 
facility.  Resources  for  conducting  the  test,  to 
include  test  equipment  and  support  personnel,  are 
generally  available  as  part  of  the  test  facility 
support  agreement.  Costs  for  this  type  of  testing 
will  be  those  associated  with  the  transportation 
of  equipment  and  systems  to  and  from  the  test 
location  and  the  TDY  costs  of  a  small  team  from 
the  JFS  or  JTF. 

In  most  instances,  lab  testing  alone  is 
insufficient  to  resolve  program  issues,  because 
the  results  generally  do  not  reflect  the  system’s 
performance  in  an  operational  environment  (i.e., 
the  system’s  performance  within  operational 
profiles  and  employed  by  operators  of  varying 
degrees  of  experience  attempting  to  accomplish 
an  operational  task).  However,  lab  testing  can  be 
a  valuable  tool  for  obtaining  operating  insights 
and  technical  data  on  a  system  prior  to  moving 
into  a  field  test  where  the  collection  of  data  is 
more  difficult.  In  this  regard,  the  data  obtained 
from  lab  tests  will  ordinarily  be  used  later  in  the 
JT&E  in  conjunction  with  data  from  other  types 
of  testing  to  address  program  issues  and,  in 
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designing  other  tests,  to  include  mini-tests  and 
field  tests. 

Modeling  and  Simulation 

Models  and  simulations  can  vary 
significantly  in  size  and  complexity  and  can  be 
useful  test  tools  in  several  respects.  They  can  be 
used  to  conduct  predictive  analyses  for 
developing  plans  for  individual  test  activities,  for 
assisting  program  planners  in  anticipating 
problem  areas,  and  for  comparison  of  predictions 
of  collected  data  and  the  results  of  discrete  tests. 
Validated  models  and  simulations  can  also  be 
used  to  examine  scenario  differences,  conduct 
what-if  tradeoffs  and  sensitivity  analyses,  and  to 
extend  test  results. 

One  sequence  that  has  proven  useful  begins 
with  model  use  to  predict  test  results.  Individual 
tests  are  then  used  to  validate  these  predictions. 
Finally,  the  validated  model  is  used  to  extend  the 
test  results  to  cases  not  tested. 

The  JFS  team  must  pay  particular  attention 
to  the  maturity,  community  acceptance,  and 
characteristics  of  models  being  considered  for 
inclusion  in  the  APA.  Model  development  is 
expensive;  similarly,  a  model  that  requires 
extensive  modification  or  completion  (most 
models  can  be  improved)  to  be  useful  may  be  too 
expensive  or  time  consuming  for  serious 
consideration. 

A  resource  for  the  JFS  to  begin 
examination  of  potential  models  applicable  to  the 
JT&E  is  the  Joint  Staff  Catalog  of  Models, 
Simulations  and  Games. 


Mini-Tests 

Mini-tests  are  small  scale  tests  involving 
one-on-one  or  one-on-few  systems  conducted 
under  operational  or  near-operational  conditions. 
They  are  used  to  collect  engineering  data,  system 
data,  procedural  data,  or  data  on  systems  where 
personnel  must  be  in  the  decision-making  loop. 
Mini-tests  are  generally  dedicated  tests  that  afford 
complete  control  by  the  JTF.  The  costs  of  mini¬ 
tests  can  vary,  based  on  factors  like  scope  and 
location  of  the  test.  Generally,  they  are  more 
expensive  than  lab  tests  but  are  considerably 
cheaper  than  field  tests.  In  some  cases, 
combinations  of  lab  tests  and  mini-tests  can 
provide  enough  data  to  satisfy  or  partially  satisfy 
program  level  issues. 

Mini-tests  have  been  used  by  JT&Es  to 
evaluate  proposed  instrumentation  schemes,  data 
collection  and  processing  techniques,  tools,  and 
procedures  for  larger  scale  field  tests.  The  use  of 
mini-tests  for  this  purpose  allows  the  team  to 
evaluate  and  revise  instrumentation,  equipment, 
and  procedures  and  train  personnel  to  perform 
the  data  collection  tasks,  thus  reducing  the  risk  of 
failure  and  the  associated  expense  (time  and 
dollars)  during  the  conduct  of  more  expensive 
testing. 

As  with  costs,  the  planning  efforts  for  a 
mini-test  can  vary  depending  on  the  scope  of  the 
test.  In  most  cases,  the  planning  efforts  are 
rather  extensive  since  most  mini-tests  are 
conducted  at  Service  test  facilities  and  ranges, 
and  require  integrating  test  facility  resources, 
JT&E  resources,  and  Service  operational  and 
support  resources. 
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Command  Post  Exercises  (CPXs) 

CPXs  are  tests  or  exercises  that  are 
primarily  procedure  oriented  and  do  not  involve 
extensive  deployment  of  forces.  For  example, 
resolving  an  issue  of  the  targeting  of  air  defense 
elements  between  ground  elements  and  tactical  air 
might  be  addressed  during  a  CPX.  Such  a  CPX 
would  involve  measurements  of  the  time  it  takes 
for  an  assembled  staff  to  receive  guidance  or 
orders,  process  targeting  information,  plan 
reactions,  coordinate  the  planning,  and  dispatch 
task  orders.  The  JFS  staff  should  evaluate 
potential  CPXs  against  the  data  required  to 
resolve  test  issues.  CPX  planning,  and 
execution  should  keep  in  mind  that  coordination 
activities  usually  occur  within  the  play  of  a  CPX, 
rather  than  before  or  more  preferably  after  it. 
Therefore,  use  of  CPXs  requires  carefully 
planned  data  collection  procedures  to  assure  that 
the  data  required  is  captured  in  the  free  play 
environment  and  that  data  collection  (by  use  of 
transparent  instrumentation)  does  not  interfere 
with  the  CPX. 

Field  Tests  and  Exercises 

Field  tests  and  exercises  are  generally  large- 
scale  activities  conducted  to  obtain  or  validate 
data  involving  system  performance  under 
operational  or  near-operational  conditions.  They 
will  range  from  one-on-one  to  many-on-many 
situations  where  data  are  required  from  many 
different  systems  interacting  simultaneously 
under  operational  or  real-world  conditions 
including  man-in-the-loop  decisions  that  impact 
the  systems’  performance  or  the  outcome  of  the 
interactions.  Field  tests  and  exercises  are  usually 
the  most  complex  and  expensive  that  a  JT&E  will 
conduct  and,  because  of  their  size  and  expense, 


will  require  extensive  planning  and  coordination 
efforts. 

The  JFS  should  examine  the  Joint  Staff 
Five  Year  Exercise  Schedule  and  Service  field 
test  and  exercise  plans  to  identify  opportunities  to 
use  scheduled  field  test  and  exercises  as  JT&E 
test  vehicles.  Criteria  for  identifying  these  test 
vehicles  should  include  test  schedule,  scenario, 
location,  forces  (red  and  blue),  systems,  primary 
objectives,  magnitude,  and  cost.  Early 
coordination  with  Joint  Staff  and  Service  field 
test  and  exercise  planners  is  necessary  for  the 
development  of  a  credible  JT&E  schedule. 

There  are  two  basic  types  of  JT&E  field 
tests,  piggybacked  and  dedicated.  Piggybacking 
refers  to  methods  for  collecting  JT&E-relevant 
data  during  already  established  Service  or  Joint 
tests  or  exercises.  This  is  distinct  from  a 
dedicated  test,  which  would  be  a  test  or  exercise 
designed  and  controlled  by  the  JT&E  for  the  sole 
purpose  of  answering  the  program  issues.  The 
costs  and  benefits  of  these  two  approaches  must 
be  weighed  in  relation  to  specific  test  goals  and 
resources.  Complete  piggybacking  may  provide 
the  lowest  cost  and  data  gathered  under  the  most 
realistic  operational  conditions.  However,  the 
inability  to  control  procedures  and  test  variables 
and  the  inability  to  iterate  or  repeat  test  events 
may  obviate  these  apparent  advantages. 

In  actuality,  there  is  a  more  desirable 
compromise  between  these  extremes.  That  is, 
conducting  JT&E  testing  in  conjunction  with  a 
field  test  or  exercise  during  a  dedicated 
"window."  During  this  window,  test  resources 
are  turned  over  to  the  control  of  the  JT&E  and 
test  scenarios  and  variables  may  be  modified  to 
meet  test  objectives.  This  method  has  proven  to 
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be  an  excellent  compromise  between  the  higher 
costs  of  dedicated  tests  and  the  need  to  exercise 
adequate  control.  The  JT&E  may  be  able  to  offer 
the  test  or  exercise  sponsors  an  incentive  for  this 
arrangement  by  providing  assets  (manpower 
equipment  and  test  results)  that  will  benefit  the 
exercise  or  lower  overall  costs. 

If  the  JT&E  only  needs  to  participate  during 
part  of  a  test  or  exercise,  or  is  going  to  implement 
testing  during  a  dedicated  window,  then  a 
decision  will  have  to  be  made  between  the  JFS 
and  the  sponsoring  agency  as  to  the  best  time  to 
conduct  JT&E  test  activities.  Piggybacking  on 


the  front  of  the  exercise  has  the  advantage  of 
having  fresher  troops  and  the  greater  possibility 
of  obtaining  additional  time  if  needed.  However, 
training  exercises  may  start  with  a  great  deal  of 
confusion  and  equipment  problems  as  well  as 
less  trained  participants.  Starting  too  late  in  the 
exercise  runs  the  risk  of  tired  or  poorly  motivated 
participants  who  are  less  than  willing  to  stay  late 
to  obtain  test  data.  Choosing  a  time  near  the 
middle  of  the  test  or  exercise  may  be  the  best  bet, 
sometime  after  things  are  expected  to  be  running 
smoothly  and  leaving  enough  time  for  additional 
testing  if  required. 
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ANNEX  G 
DESA  SUPPORT 


A.  INTRODUCTION 

DDT&E  is  responsible  to  D,T,SE&E  for 
the  overall  management  and  oversight  of  the 
JT&E  program.  This  handbook  makes  frequent 
reference  to  capabilities  that  are  available  to 
support  the  JFS.  This  annex  lists  the  specific 
DESA  capabilities  that  may  be  available. 
Support  will  be  obtained  through  the  OSD  JT&E 
Coordinator. 

B .  SPECIFIC  RESPONSIBILITIES 

•  Serves  on  the  TAB. 

•  Coordinates  disposition  of  JT&E 
resources/assets.  Provides  life-cycle 
management  of  OSD  resources  procured  by 
or  used  for  JT&E  programs. 

•  As  requested  by  the  JT&E,  provides 
technical  and  administrative  management 
services  to  JT&E  activities,  including 
staffing  and  financial  management 
assistance,  communications  and  computer 
management,  contract  management,  and 
security  and  environmental  consulting. 

•  As  requested  by  DDT&E,  provides  an 
Independent  Verification  and  Validation 
(IV&V)  of  the  technical  and  operational 
aspects  of  the  JT&E  and  conducts  data 
analysis  and  reporting. 

C .  FUNCTIONAL  SUPPORT 
CAPABILITIES 

Although  not  inclusive,  the  following 
support  capabilities  can  be  made  available  to 
support  JFS  activities.  The  JTD  should  contact 


the  JT&E  POC  for  information  relative  to 
obtaining  this  support.  If  support  is  required,  the 
FSD  should  initiate  a  Program  Identification 
Document  (PID)  or  MOA  with  DESA  to  formally 
establish  the  requirement. 

Logistics  Support 

DESA  is  the  custodian  of  Government 
investments  in  facilities,  threat  assets,  simulators, 
instrumentation,  test  equipment,  data  processing 
equipment,  and  other  peripherals  that  have  been 
purchased  by  OSD  with  JT&E  funds  and  which 
may  be  available  for  JT&E  support.  DESA 
capabilities  include  maintenance,  storage, 
configuration  management  and  control, 
warehousing,  spares  support,  inventory  control, 
transportation,  calibration,  O&M,  equipment 
training,  and  expertise  in  logistics  support  in 
general. 

Environmental  Support 

Support  in  this  area  can  cover  the  spectrum 
from  examination  of  operational  impacts  of 
testing  on  ranges  to  assistance  in  planning  a  test 
activity  to  ensure  compliance  with  NEPA 
standards  and  requirements.  This  assistance  can 
be  particularly  beneficial  when  conducting 
cost/benefit  tradeoffs  and  identifing  long  lead 
items  associated  with  Environmental 
Assessments  (EA)  and  Environmental  Impact 
Statements  (EIS).  DESA  maintains  an  extensive 
listing  of  environmental  directives  and 
regulations  and  can  provide  assistance  in  terms  of 
which  should  be  reviewed  for  applicability. 
Additionally,  DESA  can  assist  in  the 
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identification  and  acquisition  of  instrumentation 
and  display  systems  that  may  be  required  for  the 
environmental  control  of  test  activities. 

Security  Support 

Personnel  are  available  with  experience  and 
expertise  in  all  aspects  of  security  to  include 
SSO,  SCI,  SCG,  OPSEC,  COMSEC,  etc. 
DESA  support  could  include  assistance  in  the 
development  of  procedures  to  guarantee  the 
security  of  systems,  equipment,  plans, 
operations,  data,  and  reports;  obtaining  special 
security  billets  and  clearances  for  personnel;  the 
use  of  SCIF  capabilities;  the  development  of 
Security  Classification  Guides;  the  establishment 
of  interfaces  with  compartmented  projects;  and 
periodic  reviews  of  security  procedures. 

Contracting  and  Procurement  Support 

The  DESA  Director  is  head  of  a  DoD 
warranted  contracting  activity  and  the  expertise 
associated  with  this  service  is  available  to  the 
JFS.  This  expertise  includes  the  development 
and  administration  of  contracts  to  support  JFS 
activities,  the  identification  of  unique  resource 
requirements,  and  the  correlation  of  support 
requirements  with  responsible  agencies  to  assure 
that  the  required  resources  will  be  available  to 
support  planned  test  activities. 

DESA  has  existing  contracts  with  defense 
contractors  having  personnel  with  extensive 
JT&E  experience.  These  contracts  are  available 
for  use  by  a  JFS  and  a  specific  subtask  can  be 
quickly  developed  to  obtain  the  technical  support 
that  may  be  required.  If  contracting  support  is 
required,  the  FSD  will  need  to  appoint  a 
Contracting  Officers  Technical  Representative 


(COTR)  from  the  JFS  staff  who  will  be 
responsible  for  providing  technical  guidance  to 
the  contractor  and  providing  technical  inputs  on 
contractor  performance  to  the  Contracting 
Officer.  In  addition  to  providing  information  on 
and  access  to  contractors  having  JT&E 
experience  and  assistance  relative  to  contractual 
matters,  DESA  can  provide  COTR  training  for 
JFS  personnel. 

Technical  Support 

DESA  can  provide  ad  hoc  technical  support 
that  includes  commenting  on  or  contributing  to 
the  development  of  PTDs  and  any  of  their 
annexes.  Many  DESA  personnel  have  extensive 
backgrounds  in  T&E  and  work  with  and  support 
on-going  JFS  and  JTF  efforts.  Defense 
contractors  currently  under  contract  to  DESA  also 
have  personnel  with  JT&E  experience,  to  include 
past  JTF  Directors,  Service  Deputies,  Chief  of 
Staffs,  Directors  of  Operations,  and  Directors  of 
Operations  Analysis.  Technical  support  is 
basically  available  in  the  following  areas: 

•  Program  level  guidance,  assistance,  and 
advice  in  the  development  of  JFS 
documentation. 

•  Reviews  of  concepts  and  plans  developed 
by  the  JFS  and  defense  contractors 
operating  in  support  of  the  JFS. 

•  Analysis,  engineering,  and  T&E  expertise. 

•  Information  and  advice  on  databases  and 
models  that  might  be  applicable  to  the  JFS 
effort. 

•  Information  and  advice  on  range 
equipment,  test  ranges,  and  facilities. 

•  Information  and  advice  on  the  use  of  threat 
systems. 
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•  Information  and  advice  on  test 
instrumentation  systems  and 
communications  capabilities,  costs,  and 
availability. 

Personnel  Support 

DESA  cannot  themselves  provide  personnel 
on  a  permanent  or  extended  basis  to  a  JFS. 
However,  technical  assistance  as  described  in  the 
previous  paragraph  can  be  made  available  on  an 
occasional  basis.  Additionally,  DESA  can  assist 
the  JFS  in  developing  and  coordinating 
manpower  requirements  with  the  Services  and 
the  Civil  Service  in  requisitioning  such 
manpower.  DESA  can  also  assist  in  providing 
contractual  support  from  their  pool  of 
Government  contractors. 


Accounting  and  Finance  Support 

Accounting  and  finance  expertise  is 
available  and  can  be  provided.  DESA  personnel 
are  familiar  with  the  accounting  and  finance 
procedures  and  peculiarities  of  all  the  Services 
and  can  be  helpful  to  the  JFS  in  establishing 
liaisons  with  the  respective  Service  finance 
centers.  DESA  can  also  provide  advice  on 
Service  procedures  and  periodic  reviews  of  JFS 
fiscal  situations  and  reports  to  assure  their 
accuracy.  However,  dedicated  DESA  finance 
and  accounting  personnel  are  not  available  to  any 
individual  JFS  or  JT&E. 
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ANNEX  H 

TEST  RANGES  AND  FACILITIES 


A.  INTRODUCTION 

The  JT&E  program  schedule  is  based  on 
early  JFS  coordination  with  OSD,  Joint  Staff, 
and  the  Services  relative  to  the  use  of  test  ranges 
or  facilities.  The  JFS  must  be  aware  of  and 
consider  the  coordination  process  that  the  JTF 
must  undertake  to  formalize  the  use  of  test  ranges 
and  facilities  that  are  identified  in  the  APA. 

The  JTF  must  formalize  the  requirement  for 
use  of  these  ranges/facilities  with  the 
range/facility  managers.  The  coordination  for 
use  of  a  test  range  or  facility  should  be  in  the 
form  of  a  Program  Introduction  Document  (PID). 
Although  range  and  facility  use  is  included  in  the 
Consolidated  Resource  Estimate  (CRE),  the 
range/facility  manager  will  take  no  action  on  this 
requirement  until  a  PID  is  received.  PIDs  are 
completed  using  the  instructions  in  Universal 
Documentation  System  (UDS)  Document  501-79 
and  the  supporting  agency  instructions.  The 
range/facility  manager  will  provide  the  JTF  with 
a  statement  of  capability  as  acceptance  of 
responsibilities  as  outlined  in  the  PID. 

The  PID  should  be  based  on  a 
comprehensive  survey  of  the  test  range  or 
facility.  The  survey  data  and  information  will 
form  the  basis  for  detailed  test  planning  and 
decision  making.  During  the  field  portion  of  the 
survey,  the  survey  team  should  focus  on 
technical,  operational,  and  logistic  considerations 
required  for  planning  the  support  necessary  to 
execute  the  planned  test  activity  or  event. 
Circumstances  may  be  encountered  that  require 
follow-on  visits  to  investigate  or  evaluate 
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potential  problem  areas.  The  survey  team  should 
take  the  necessary  time  to  do  the  job  right. 

The  survey  team  should  review  the  current 
versions  of  the  following  documentation  prior  to 
beginning  the  survey: 


1.  Describe  the  range  or  facility  in  detail. 
Provide  the  geographical  location, 
topography,  noncontiguous  aspects,  and 
climatic  conditions  associated  with  the  range 
or  facility. 

2.  Provide  the  size  and  dimensions  of  the  range 
or  facility  and  identify  any  restricted  land  or 

H-l 


•  DoD  Directive  3200.11 -D  Major  Range  and 
Test  Facility  Base  Summary  of  Capabilities. 

•  Report  on  Range  Utilization  and 
Improvement  Plans. 

•  USAF  Armament  Division  Technical 
Facilities  Manual,  Volumes  I,  II,  and  III. 

•  US  Army  Test  Facilities  Register,  Volumes  I 
and  II. 

•  Test  and  Range  Facilities  Catalog  -  Naval 
Sea  Systems  Command 

•  Naval  Test  Facilities  Data  Base  (access 
through  Technet) 

The  range  or  facility  survey  should  acquire 
or  develop  data  and  information  on  the  following 
test  range  or  facility  capabilities  and  should 
conclude  with  positive  and  well  documented 
recommendations  on  which  planners  and 
managers  can  make  decisions  or  develop  plans 
and  procedures. 

B  .  TEST  RANGE  OR  FACILITY 


air  space  usage  requirements.  Include  maps 
and  technical  documentation  when 
appropriate. 

3.  Address  the  range  or  facility  operating 
schedule  and  availability  on  a  yearly  basis, 
closure  conditions,  EMI/EMC  impacts,  and 
proximity  of  the  area  to  on  and  off  range 
military  and  commercial  flight  patterns. 

4.  Address  the  availability  of  floor  space  in 
existing  structures  that  can  be  used  by  JTF 
personnel. 

5.  Identify  the  range  or  facility  management 
hierarchy  and  directives  for  management, 
conduct,  and  support  of  test  activities. 

C .  LOGISTICS 

1 .  Test  range  or  facility  accessibility. 

•  Air-  Provide  name  and  location  of 
airfields,  military  and  commercial. 
Identify  servicing  airlines. 

•  Roads-  Identify  intra-range  or  facility 
roads  to  primary  highway  and 
commercial  highway  transportation 
facilities. 

•  Water  -  Identify  port  or  access 
waterways,  distances  to  the  range  or 
facility,  restrictions  to  use,  and  suitability 
for  test  activity  or  event  use. 

2.  Nearest  Military  Establishment.  Identify  the 
nearest  military  establishment  that  can 
provide  logistic  and  supply  support. 
Establish  POC  and  determine  the 
administrative  vehicle  needed  to  acquire 
service  support  to  include: 

•  Billets  for  Officers,  enlisted  and 
contractors. 

•  Base  Exchange. 

•  Clothing  Sales. 


•  Commissary. 

•  Concessions  (barber  shop,  laundry, 
movies,  etc.). 

•  Dining  Facilities. 

•  Medical  Services. 

•  POL  (to  include  delivery  to  test  sites). 

•  Reference  Library  (technical). 

•  Religious  Services. 

•  Secure  Storage. 

•  Supply  Support. 

•  Transportation  (passenger  and  freight). 

•  Security  (guard  service). 

D.  CIVIL  ENGINEERING 

1.  Determine  the  level  of  support  that  civil 

engineers  are  able  to  provide  and  the 

response  times  associated  with  the  support. 

2.  Determine  the  type  of  electrical  service  that  is 

available  at  test  sites  to  include: 

•  Prime  power  or  generator  power. 

•  Capacity  of  power  plant  to  include 
voltage,  frequency,  and  phasing. 
Determine  size  of  power  plant  and 
availability  of  spare  service. 

•  Determine  if  back  up  power  is  available 
and  is  start  up  manual  or  automatic. 

•  Determine  if  grounding  and  lightning 
protection  is  available  for  range  or  facility 
systems. 

E.  SECURITY 

Determine  the  existence  of  test  range  or 

facility  security  plans  and  check  the  following: 

•  Area  security  provisions  and  assigned 
priorities  for  their  protection. 

•  Control  measures  and  established 
restrictions  to  access  and  movement  into 
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critical  areas  (personnel,  vehicles,  and 
materials). 

•  Determine  aids  to  security,  such  as 
protective  barriers,  signs,  gates, 
protective  lighting  systems, 
communications,  and  the  availability  of 
security  forces. 

•  Determine  if  contingency  plans  are 
responsive  to  emergency  situations. 

•  Determine  the  requirements  for  passing 
clearance  data  on  personnel  for  range  or 
facility  access. 

F.  SAFETY 

Establish  contact  with  the  range  or  facility 
safety  officer  and  obtain  copies  of  range  or 
facility  safety  directives,  brochures,  POCs,  and 
key  telephone  numbers.  Also  obtain  information 
on  the  following: 

•  Range  or  facility  speed  limits. 

•  Use  of  rental  cars. 

•  '  Identify  the  meanings  of  different  colored 

warning  lights. 

•  Identify  precautions  required  at  remote 
test  sites. 

•  Identify  criteria  for  evacuation  of  range  or 
facilities. 

•  Determine  locations  of  first  aid  stations. 

•  Identify  restricted  or  hazardous 
operations  areas. 

•  Identify  methods  of  notification  for  range 
or  facility  closure. 

G .  RANGE  OR  FACILITY  SYSTEMS 

AND  EQUIPMENT 

Range  and  facility  systems  and  equipment 
are  normally  identified  in  master  planning 


documents  which  are  afforded  an  appropriate 
level  of  classification.  The  JTF  survey  team 
should  obtain  copies  of  these  documents,  if 
feasible.  If  not,  collect  the  data  listed  in  the 
following  paragraphs.  Identify  the  equipment  or 
systems,  determine  the  quantity  on  hand,  and 
obtain  their  authorized  frequencies,  operational 
locations,  and  geographical  coordinates.  Some 
of  the  frequencies  may  be  classified. 

•  Communications.  Determine  if  the 
required  frequency  spectrum  is 
authorized/used. 

•  Targets. 

•  Range  or  Facility  Instrumentation. 
Identify  the  data  source  mechanization, 
the  end  product,  and  fidelity  of  the 
available  data  sources.  Determine  the  data 
formats  and  elements  for  each  source 
product.  Determine  if  the  systems  are 
certified  prior  to  the  start  of  missions  and 
if  the  system  modifications  are  controlled. 

(1)  Scoring  Systems. 

(2)  Acquisition  Systems. 

(3)  Threat  Systems. 

H .  RANGE  OR  FACILITY  SUPPORT 
SYSTEMS 

I .  Range  Timing  System. 

•  Identify  the  time  code  format  and  timing 
technique  used. 

•  Determine  if  system  is  available  to  all  test 
sites. 

•  Determine  compatibility  with 
instrumentation  requirements. 

•  Determine  availability  of  backup  source. 

•  Determine  time-synch  capabilities. 
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2.  Integrated  Air  Defense  Systems. 

•  Determine  if  systems  are  available. 

•  Determine  if  systems  are  automatic,  semi¬ 
automatic,  or  manual. 

•  Determine  system  hardware 
configurations. 

3.  Intra-base  Radios. 

•  Determine  nets  that  are  available  and  their 
respective  functions. 

•  Determine  types  of  equipment  and 
authorized  frequencies. 

•  Determine  the  availability  of  radios  for 
white  communications  use. 

4.  Range  or  Facility  Calibration  Systems. 

•  Define  system  configuration/network. 

•  Determine  if  systems  are  integrated  with 
timing  system. 

•  Determine  if  calibration  models  and 
databases  of  historical  information  are 
available. 

5 .  Automatic  Data  Processing. 

•  Define  the  computer  systems  and  the 
companion  data  collection  systems. 

•  Describe  the  computer  hardware  interface 
with  the  data  sources. 

•  Determine  if  results  of  test  are  provided 
on  a  real-time  basis. 

•  Identify  and  obtain  copies  of  computer 
products. 


I.  DISPLAY  SYSTEMS 

1.  Identify  the  type,  location,  and  function  of 
display  systems. 

2.  Identify  computer  programs  in  the  systems, 
and  the  communications  interfaces  that  accept 
and  feed  the  raw  data  to  the  computers. 

3.  Determine  if  the  display  systems  are  on  a 
real-time  or  near  real-time  basis.  Determine  if 
systems  have  capability  to  produce  hard 
copies  of  displayed  data. 

J.  FINANCIAL 

Identify  the  point  of  contact  and  the 
established  arrangements  for  user  reimbursement 
of  costs  associated  with  the  use  of  the  range  or 
facility  and  options  for  cost  sharing  of 
improvements  that  might  be  required. 

K.  SUMMARY 

Provide  an  overall  assessment  of  the  range 
or  facility,  the  anticipated  support,  and  any 
details  that  might  contribute  to  test  planning  and 
execution. 

L.  RECOMMENDATIONS 

Be  specific.  If  the  survey  teams  discover 
problems  that  could  seriously  impact  test 
execution,  the  report  should  so  state  and 
document  the  finding  or  identify  the  specific 
limitations  that  use  of  the  range  or  facility  would 
have  on  test  execution. 
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ANNEX  I 

CONSOLIDATED  RESOURCE  ESTIMATE  (CRE)  OUTLINE 


The  CRE  is  prepared  to  consolidate  JT&E 
resource  requirements  into  a  single  document. 
The  CRE  documents  JFS  projections  of  JT&E 
resource  requirements  and  will  subsequently  be 
used  as  the  basis  for  the  initiation  of  Service 
resource  requests  (TRPs  and  OTPs).  The 
following  CRE  format  is  extracted  from  the 
Operational  Test  Agencies'  (OTA)  MOA  for 
MOT&Es  and  JT&Es. 

1.  Tide 

2.  References:  Nominating  Organization, 
Charter,  Dates 

3 .  Purpose  of  JT&E 

4 .  Scope  and  Tactical  Relationships 

5 .  Test  Concepts  or  Issues/Objectives 

6.  Lead/Participating  Services 

7.  Services  POC  Lists 

8 .  Proposed  Test  Installation  Locations 

9 .  Test  Activity  Dates 

1 0 .  Test  Directorate  Personnel/Equipment 

a.  Test  Staff 

1)  Data  Management 

2)  Logistical 

3)  Administrative 

4)  Test  Operations 

5)  Controllers 

6)  Data  Collectors 

7)  Software  Evaluators 

b.  Aviation  Support 


c.  Signal/Communications 

d.  Miscellaneous  Equipment 

e.  Training  Requirements 

1 1 .  Player  Participants  Personnel/Equipment 

a.  Blue  Force 

1)  Ground  Players/Units 

2)  Aviation  Players/Units 

3)  Ground  Players  Equipment 

4)  Aircraft  Hours/Types 

5)  Training  Requirements 

b.  Red  Force 

1)  Ground  Players/Units 

2)  Aviation  Players/Units 

3)  Ground  Players  Equipment 

4)  Aircraft  Hours/Types 

5)  Training  Requirements 

12.  Installation  Support 

13.  Test  Targets 

14.  Instrumentation 

15.  ADP 

16.  Ammunition/Missiles 

17.  POL 

1 8 .  Contractor  Support 

19.  Funding  Estimates 

20.  Milestones 

2 1 .  Test  Range  Support 

22.  Computer  Simulators/Models/Test  Beds 

23.  Threat  Systems/Surrogates/SIMS 

24 .  Foreign  Material  to  Replicate  the  Threat 

25.  Accreditation  Support 
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ANNEX  J 

TEST  PROGRAM  COST/BUDGET  MATRIX 


Joint  Feasibility  Studies  are  required  to 
develop  a  Consolidated  Resource  Estimate  (CRE) 
and  a  by-year  budget  projection  if  recommending 
the  chartering  of  a  Joint  Test  and  Evaluation 
(JT&E).  The  attached  test  program 
cost/budget  matrix  illustrates  how  these 
resource  requirements  can  be  translated  into 
funding  requirements.  The  matrix  attempts  to 
capture  the  primary  JT&E  activities  that  have 
associated  costs,  the  type  of  activities  that  need  to 
be  examined,  and  methods  that  could  be  used  to 
estimate  the  dollars  required  to  support  these 
particular  activities.  Once  the  costs  have  been 
determined  they  are  aggregated  to  arrive  at  the 
CRE  and  projected  yearly  budget  estimate.  The 
matrix  is  generic  in  nature  and  will  need  to  be 
adjusted  to  fit  the  specific  requirements  of  each 
proposed  JT&E.  Some  JT&Es  will  have  unique 
testing/activity  requirements  not  reflected  by  the 
topics  in  the  matrix  and  others  will  have  no  need 
for  some  of  the  topics  listed. 


In  general,  things  to  consider  in  filling  out 
the  matrix  for  a  particular  JT&E  include  the 
following: 

•  Aircraft/ship/platform  by  type/sorties. 

•  Weapon(s)  by  type. 

•  Communication  equipment  by  type  and 
mission  (operational  and  test  support). 

•  Target(s). 

•  Characterization  requirements. 

•  Instrumentation  (beacon,  pod,  TSPI,  GPS, 
TACTS,  ACMI,  GRDCUS,  VTR,  War 
wagons,  etc.). 

•  Meteorological  requirements. 

•  Modeling/simulations. 

•  Data  sources/devices  (9  track,  floppy  disks, 
Bernoulli,  bar  code  labels,  etc.). 

•  Data  collectors  (government,  civilian, 
military,  contractors). 

•  Operational  equipment. 

•  Test  unique  equipment  (equipment 
consumed  during  the  JT&E  or  no  longer 
required  after  the  JT&E). 
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PRIMARY  ACTIVITIES  COST  CATEGORY 


DIRECT  TEST  COSTS 

TEST  PREPARATION  Environmental  Impact  Study 

Note:  Preparation  costs  must  be  Safety  Plan 

prepared  for  each  test  site/event.  _ 

Train/Coordinate  Test 
Instrumentation  Operators 


METHOD/EXAMPLE 


number  of  operators 
travel  to  training  site 
per  diem  during  travel  and 
TDY 

rental  cars 


Test  Communication  Network 


TEST  ASSETS 


Test  Specific  Materials 


Note:  These  are  materials 
prepared  for  each  test  site/event 
and  be  will  be  unique  for  each 
JTF.  Examples  are  from  JCCD. 


Instrumentation 


Projected  TDY  Costs 


Test-Related  Contract  Support 


packaging  and  shipping 
communication  network 
setup 

travel  to  test  site 

per  diem  during  travel  and 

TDY 

rental  cars _ 

smoke  generators 
remote  controls 
woodland,  snow,  desert 
camouflage 
screens/supports 
runway  false  operating 
surface 

decoy  buildings  and  piers 
radar  comer  reflectors 
photo  image  canvas 
expendables  (e.g.  fog  oil, 
tonedown  paint,  RAM/IR 
coatings,  packaging  & 

shipping  costs) _ 

rental/lease  of  instruments 
calibration  costs 
packaging  and  shipping 
operator  costs  if  not 
included  elsewhere  (e.g. 
pay  &  allowances,  travel, 
per  diem,  rental  car) 
transportation/travel  (for 
each  site/person  to/from 
each  home  station) 

per  diem  (per  person  per 
day  for  number  of  days) 

rental  car  cost  (number  of 
cars,  projected  days, 
average  cost  per  day) 

JTF  on-site  support 
test  site/event  support 


J-2 


JFS  Handbook 
February  1, 1996 


PRIMARY  ACTIVITIES 

COST  CATEGORY 

METHOD/EXAMPLE 

TEST  ASSETS 
(Cont) 

Range  Costs 

-  range-owned 
instrumentation 

-  range  control/data 
processing 

-  range  communication 
costs 

-  range  utilization 

-  range  safety/fire/medical 
emergency  crew 

Pretest  Modeling  Costs 

-  development  of  input 
parameters  and 
characteristics  to  run  the 
model 

-  model  modification  cost 

-  projected  cost  to  perform 
modeling 

Dry  Runs 

-  execution  of  unique  trials 

TEST  DATA  PROCESSING  COSTS 

DATA  COMPILATION 

Note:  Prepared  for  each  test 
site/activity.  It  includes  the 
physical  handling  of  the  data, 
input,  documenting  calibration 
procedures  and  instrumentation 
characteristics,  obtaining  data 
samples,  etc.) 

Manpower  Required  (man 
days/months) 

-  targets 

-  participants  (e.g.  aircraft, 
tanks,  ships,  etc.) 

-  meteorological 

Test  Range  Data  Retrieval 
Reduction 

DATA  REDUCTION  AND 
RE-COMPILATION 

Note:  Prepared  for  each  test 
site/event.  It  includes  data 
reduction  and  re-compilation 
processing  of  data,  checking 
range  to  target  data,  validating 
calibrations,  assimilating  data 
from  similar  instruments  at 
different  sites/events 

Manpower  Required  (man 
days/months) 

-  per  instrumentation 
requirement 

DATA  ANALYSIS 

Note:  Prepared  for  each  test 
site/activity. 

Manpower  Required  (man 
days/months) 

-  per  instrumentation  or 
test/activity  requirement 
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PRIMARY  ACTIVITIES 

COST  CATEGORY 

METHOD/EXAMPLE 

DATA  COMPARISON 

Note:  Multi-treatment/cross¬ 
platform  comparisons 

Manpower  Required  (man 
days/months) 

DATA  ARCHIVE 
PREPARATION 

Note:  Prepared  for  each  test 
site/activity. 

Manpower  Required  (man 
days/months) 

COMPARE/ANALYZE 
PREDICTED  TO  ACTUAL 
TEST  RESULTS 

Manpower  Required  (man 
days/months) 

ANALYZE 

COMPARATIVE  TEST 
RESULTS 

Note:  Multi-tests/activities. 

Manpower  Required  (man 
days/months) 

TEST-UNIQUE  COMPUTER  HARDWARE/SOFTWARE  COSTS  (AFTER  USE 
ITEMS  ARE  EITHER  CONSUMED  OR  ARE  NO  LONGER  REQUIRED) 

TEST-UNIQUE 
COMPUTER  HARDWARE 

Workstations 

-  type(s)  of  station 

-  cost/numbers 

High  Resolution  Window 
Terminals 

-  type(s)  of  terminals 

-  cost/numbers 

Portable  Computer  Hardware 

-  type(s)  of  computers 

-  cost/numbers 

Server 

-  type(s)  of  server 

-  cost/numbers 

TEST-UNIQUE  STORAGE 
DEVICES 

Tape  Drives 

-  type(s)  of  tape  drives 

-  cost/numbers 

Disk  Drives 

-  type(s)  of  disk  drives 

-  cost/numbers 

TEST-UNIQUE  OTHER 
HARDWARE 

Laser  Printers 

-  type(s)  of  laser  printers 

-  cost/numbers 

Color  Printers 

-  type(s)  of  color  printers 

-  cost/numbers 

Dot  Matrix  Printers 

-  type(s)  of  dot  matrix 
printers 

-  cost/number 

Other  Printers 

-  type(s)  of  other  printers 

-  cost/number 
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PRIMARY  ACTIVITIES 

COST  CATEGORY 

METHOD/EXAMPLE 

TEST-UNIQUE 

COMPUTER 

MAINTENANCE 

Network/Hardware 

Management 

-  manpower  required  (man 
months) 

Hardware  Maintenance 

TEST-UNIQUE 

COMMERCIAL 

SOFTWARE 

Licensed  Commercial  Software 

-  type  of  software 

-  cost/numbers  of  copies 

DATABASE-RELATED 

COMPUTER  HARDWARE/SOFTWARE  COSTS 

DATABASE-RELATED 
COMPUTER  HARDWARE 

Workstations 

-  type  of  workstations 

-  cost/number  of 
workstations 

Computer  Terminals 

-  type  of  terminals 

-  cost/number  of  terminals 

Server(s) 

-  type  of  servers 

-  cost/number  of  servers 

DATABASE-RELATED 
STORAGE  DEVICES 

Tape  Drive(s) 

-  type  of  tape  drives 

-  cost/number  of  tape  drives 

Optical  Disk(s) 

-  type  of  optical  disks 

-  cost/number  of  optical 
disks 

Disk  Drive(s) 

-  type  of  disk  drives 

-  cost/number  of  disk  drives 

DATABASE-RELATED 
OTHER  HARDWARE 

Laser  Printers 

-  type(s)  of  laser  printers 

-  cost/number  of  laser 
printers 

Color  Printers 

-  type(s)  of  color  printers 

-  cost/number  of  color 
printers 

Dot  Matrix  Printers 

-  type(s)  of  dot  matrix 
printers 

-  cost/number  of  dot  matrix 
printers 

DATABASE-RELATED 

MAINTENANCE 

Hardware  Maintenance 

DATABASE-RELATED 

SOFTWARE 

Licensed  Software 

-  type  of  software 

-  cost/number  of  copies 

Purchased  Software 

-  type  of  software 

-  cost/number  of  copies 

Developed  Software 

-  type  of  software 

-  projected  cost/manpower 

Other  Software 
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COST  CATEGORY 


PRIMARY  ACTIVITIES 


OTHER  CONTRACT  SUPPORT  COSTS 


METHOD/EXAMPLE 


TEST  DESIGN  SUPPORT 

Manpower 

-  cost  per  man 
day/month/year 

TEST  PLANNING 
SUPPORT 

Manpower 

-  cost  per  man 
day/month/year 

REPORT  PREPARATION 

Manpower 

-  cost  per  man 
day/month/year 

DEVELOP/REFINE 

MODEL  CALIBRATION 
TOOLS 

Manpower 

-  cost  per  man 
day/month/year 

OTHER  COSTS 

NON-TEST  EVENT 
RELATED  TRAVEL 

Conferences/Technical 

Interchanges 

-  number  of  trips 

-  average  cost  (average 
length,  max.  per  diem, 
rental  car,  average 
transportation  cost) 

Administrative  Travel 
(IPR,  TAB,  GOSC,  etc.) 

-  number  of  trips 

-  average  cost  (average 
length,  max.  per  diem, 
rental  car,  average 
transportation  cost) 

LEAD  AND  PARTICIPATING  SERVICE  COSTS  (NON-OSD) 
Civilian  Pay 


Office  Facilities 

Vehicle  Rental 

TDY  and  Per  Diem  (non-JTF) 

Base  Support  (ISSA) 

Platforms  and  Equipment 

-  Flying  Hours 

-  POL 

-  Munitions 

Furniture 

Non-test  Specific  Supplies  and 
Equipment 
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ANNEX  K 

ENVIRONMENTAL  CONCERNS 


A.  INTRODUCTION 

JFS  planning  for  the  execution  of  the  JT&E 
must  consider  the  environmental  effects  of  test 
activities  in  the  development  of  the  test  concept 
and  APA.  Failure  to  consider  environmental 
compliance  requirements  could  result  in  negative 
consequences  to  include  rejection  of  JFS 
recommendations,  test  activity  cancellation,  or 
unacceptable  schedule  delays  after  the  JTF  is 
chartered.  Avoiding  these  consequences  requires 
that  the  JFS  understand  and  develop  an 
appreciation  of  the  National  Environmental 
Policy  Act  (NEPA)  and  associated  DoD 
Directives. 

It  is  important  when  considering  a  JT&E 
test  activity  to  be  aware  that  DoD  is  the  Federal 
leader  in  agency  environmental  compliance  and 
protection.  There  are  more  than  40  Federal 
environmental  statutes  which  could  affect 
contemplated  JT&E  test  activities.  State  and  local 
environmental  laws  are  derived  from  these 
Federal  regulations  and  are  usually  more 
restrictive. 

This  annex  provides  information  on 
environmental  issues  and  procedures  that  might 
apply  to  the  JT&E.  If  test  activities  under 
consideration  are  to  be  conducted  on  an 
established  DoD  facility  or  major  range  test 
facility,  coordination  with  local  environmental 
personnel  and  compliance  with  established  local 
procedures  should  be  adequate. 

Most  military  or  Government  test  facilities 
have  their  own  environmental  departments  that 


have  completed  many  environmental  impact 
studies  and  subsequent  statements  based  on 
previous  test  activities.  These  environmental 
impact  statements  can  be  used  provided  the 
proposed  JT&E  test  activity  has  similar  assets 
and  scenarios  of  past  test  activities. 

If  the  JT&E  is  considering  test  activities  on 
non-DoD  lands  or  test  activities  that  will  require 
the  transport  of  equipment  or  materials  over 
public  transportation  systems,  the  JFS  must 
become  familiar  with  and  consider  the  possible 
impact  of  environmental  directives  and 
regulations  for  the  areas  of  interest.  These 
directives  and  regulations  can  be  obtained  by 
contacting  the  Environmental  Protection  Agency 
(EPA)  Regional  Offices.  The  EPA  can  also 
provide  assistance  relative  to  state  and  local 
regulations  for  the  area  of  interest. 

DESA  can  provide  a  wide  range  of  help  to 
the  FSD  to  ensure  that  environmental  issues  are 
properly  considered.  Specific  help  includes  legal 
counseling,  access  to  environmental  planning 
tools,  databases,  and  expertise  and  advice  on 
obtaining  special  instrumentation  that  might  be 
required  to  monitor  environmental  compliance. 

B  .  TEST  ACTIVITY  CONSIDERA¬ 
TIONS 

The  environmental  challenge  for  any  test  is 
the  identification  of  proposed  test  activities  that 
might  produce  adverse  biological,  ecological,  or 
socioeconomic  environments.  Some  test  activities 
will  have  no  impact  on  these  areas.  In  others,  the 
projected  impact  might  range  from  some  to 
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severe.  In  either  case,  the  planned  test  activities 
must  comply  with  environmental  laws  and 
regulations.  Two  basic  tenants  of  these  laws  and 
regulations  are: 

•  Test  activity  procedures  must  ensure  that 
environmental  information  is  available  to 
decision  makers  and  citizens. 

•  Test  activity  plans  must  consider  reasonable 
alternatives  to  avoid  or  minimize 
environmental  adverse  effects. 

Environmental  regulations  are  classified  as 
either  procedural  or  substantive.  The  NEPA 
requires  all  Federal  agencies  to  consider 
environmental  impacts  during  the  decision 
making  process  regarding  proposed  test 
activities.  Depending  on  whether  a  proposed  test 
activity  could  significantly  affect  the 
environment,  one  of  these  levels  of  analysis  is 
required:  (1)  Categorical  Exclusions  (CATEX), 
(2)  Environmental  Assessments  (EA),  or  (3) 
Environmental  Impact  Statements  (EIS).  The 
JFS  should  include  estimates  for  the  time  and 
expense  that  might  be  required  for  these 
environmental  considerations  for  each  of  the 
proposed  test  activities. 

Categorical  Exclusions  (CATEX) 

Categorical  Exclusions  are  those  test 
activities  that  do  not  normally  have  a  significant 
individual  or  cumulative  effect  on  the 
environment  or  have  previously  been  found  to 
have  no  such  effect  as  a  result  of  procedures 
adopted  by  the  DoD.  Categorical  Exclusion 
activities  are  listed  in  DoD  Directive  6050.1  and 
they  do  not  require  an  EA  or  EIS. 


Environmental  Assessments  (EA) 

An  EA  is  an  analysis  of  the  potential 
environmental  impact  of  a  proposed  test  activity. 
An  EA  may  be  required  when  the  JFS  can  not 
determine  beforehand  whether  the  test  activity 
under  consideration  will  affect  the  environment 
or  will  be  controversial  with  respect  to 
environmental  effects.  If  required,  the  EA  will 
either  conclude  that  the  test  activity  will  not 
significantly  effect  the  environment,  thus 
resulting  in  the  preparation  of  a  Finding  of  No 
Significant  Impact  (FONSI)  or  will  conclude  that 
the  test  activity  could  have  a  significant  impact. 

Environmental  Impact  Statements  (EIS) 

The  JFS  should  identify  test  activities  that 
could  involve  potential  environmental  impacts. 
The  environmental  impact  of  proposed  test 
activities  should  be  considered  in  the  selection  of 
the  test  method  and  the  feasibility  of  conducting 
the  test  activities  at  specific  locations.  If 
environmental  impact  is  anticipated  and  there  is 
no  alternative  test  method,  the  JTF  may  be 
required  to  conduct  an  EIS.  The  following  are 
characteristics  of  an  EIS: 

•  An  EIS  is  a  formal  consideration  of 
environmental  consequences.  It  addresses 
the  nature  of  the  action  and  its  potential 
impact  on  the  environment.  The  EIS  must 
address  all  measures  that  can  and  will  be 
taken  to  minimize  these  impacts  and 
alternatives  to  the  proposed  test  activity,  to 
include  the  consequences  of  not  doing  the 
entire  test  activity. 

•  An  EIS  is  both  complex  and  lengthy.  It  is 
possible  to  have  the  EIS  prepared  by  a 
contractor. 
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•  The  length  and  cost  of  an  EIS  can  be 
extensive  and  may  require  several  man- 
years  of  specialized  analytical  effort  to 
complete.  Thus,  the  requirement  for  the 
identification  of  a  potential  EIS  requirement 
for  proposed  JT&E  test  activities  cannot  be 
overemphasized. 

C.  ENVIRONMENTAL  IMPACT  CON¬ 
SIDERATIONS 

The  JFS  should  as  a  minimum  include  the 
following  in  their  consideration  of  environmental 
impacts: 

•  Determine  test  activity  scenario,  location, 
assets,  environmental  concerns,  and 
considerations. 

•  Determine  if  test  facility  has  environmental 
impact  statements  that  would  be  applicable 
to  the  proposed  test  activity.  The  test 
facility  managers  will  know  what 
requirements  are  fulfilled  or  need  further 
study. 

•  Determine  what  environmental  impact 
actions  are  ongoing. 

•  If  environmental  actions  will  be  required, 
estimate  associated  cost  and  time  to 
accomplish. 

•  Consider  other  test  activity  options. 

D.  TEST  EVENT  CONSIDERATIONS 

The  following  is  a  list  of  some  test  and 
evaluation  considerations  that  need  to  be 
addressed  by  the  JFS: 

•  Weapon  Firing  (noise  and  air  emissions). 

•  System  maintenance  (waste  solvents  and 
oils). 


•  Vehicle  operation  (noise,  terrain  impacts, 
and  air  emissions). 

•  Contamination  of  land  resources  and 
unexplored  ordinance. 

•  Accidents  involving  hazardous  materials. 

•  Test  site  cleanup. 

•  Land  use  management. 

•  Wildlife  protection. 

•  Archaeological  and  historical  resources. 

E.  PUBLIC  INVOLVEMENT 

Public  participation  in  preparing  EAs  and 
EISs  is  a  reality  that  the  JFS  will  have  to 
consider.  Therefore,  should  a  test  activity  involve 
the  potential  for  a  significant  environmental 
impact,  the  JFS  should  consider  the  requirement 
that  the  JTF  will  have  to  establish  appropriate 
communications  with  local  authorities  and 
interested  parties.  In  determining  the  extent  to 
which  public  participation  could  be  required,  the 
JFS  should  consider  the  following  factors: 

•  The  magnitude  of  the  environmental 
considerations  associate  with  the  proposed 
test  activity. 

•  The  extent  of  anticipated  public  interest. 

•  Relevant  questions  of  national  security  and 
classification. 

F.  CLASSIFIED  TEST  ACTIVITIES 

Classified  tests  must  comply  with  the  same 
environmental  laws,  regulations,  and  directives 
as  unclassified  tests.  EISs  and  EAs  will  have  to 
be  prepared,  safeguarded,  and  disseminated 
according  to  the  requirements  applicable  to  all 
classified  information.  When  feasible,  the 
documents  should  be  organized  in  such  a  manner 
that  classified  portions  are  included  as 
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appendices,  so  that  the  unclassified  portions  may 
be  made  available  to  the  public. 

A  classified  EA/EIS  serves  the  same 
informed  decision  making  purpose  as  do  the 
unclassified  versions.  Even  though  the  classified 
EA/EIS  does  not  undergo  public  review  and 
comment,  it  will  be  part  of  the  information 
package  for  each  JT&E  activity.  The  content  of  a 
classified  EA/EIS  must,  therefore,  meet  the  same 
content  requirement  that  is  applicable  to  a 
published  unclassified  version. 

G.  JTF  FOREIGN  TEST  ACTIVITIES 

The  environmental  situation  of  a  joint  test 
conducted  outside  the  territories  of  the  United 
States  will  depend  on  the  requirements  and 


standards  set  by  the  host  and  Status  of  Forces 
Agreement  (SOFA).  Thus,  if  a  test  activity  is 
considered  at  a  non-U.S.  location,  the  JFS  must 
be  familiar  with  the  specific  regulations  that  apply 
to  testing  in  the  host  country  and  the  specific 
locale  within  that  country.  All  U.S.  facilities 
located  outside  the  territories  of  the  United  States 
are  required  to  conduct  their  test  activities  in  an 
environmentally  safe  manner  and  in  compliance 
with  both  host  country  and  U.S.  military 
regulations  relating  to  environmental,  natural 
resource  protection,  and  occupational  health  and 
safety.  The  two  major  U.S.  directives  that  apply 
to  or  specifically  address  testing  issues  are: 

•  DoD  Directive  4120.1 4. 

•  EO  12114:  Environmental  Effects  Abroad 
of  Major  Federal  Actions. 
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ANNEX  L 

SECURITY  CLASSIFICATION  GUIDE 


If  the  JFS  involves  the  development  of 
classified  material,  the  JFS  should  consider  the 
requirement  to  develop  a  Security  Classification 
Guide  (SCG).  The  following  is  an  example  of 


an  SCG  format  developed  by  a  recent  JTF  that 
could  be  tailored  to  the  specific  requirements  of 
the  JFS. 


A.  COVER 

Title: 

Issued  By: 

Approved  By: 

Date: 

Action  Officer: 

Distribution  Notice:  This  document  covers  the  operational  use  of  military  systems  and 

hardware.  Distribution  restricted  to  DoD  and  DoD  contractors.  Further 
dissemination  only  as  directed  by  JXXX/JFSD,  XX  Road,  Any  Town, 
USA. 


B.  SECTION  I 

GENERAL  INSTRUCTIONS 

1 .  PURPOSE.  To  provide  instructions  and 
guidance  on  the  classification  of  information 
developed  by  the  JXXX/JFS. 

2 .  AUTHORITY 

a.  This  guide  is  issued  under  the  provisions 
of  Executive  Order  12356;  DoD 
5200.1R,  Information  Security  Program 
Regulation;  DoD  5220.22M;  and  DoD 
5220.22R. 

b.  This  guide  constitutes  the  authority  for, 
and  may  be  cited  as  the  basis  for, 


classification,  regarding,  or 
declassification  of  information  and 
material  originated  by  the  JXXX/JFS. 

3.  OFFICE  OF  PRIMARY 
RESPONSIBILITY  (OPR).  All 

inquiries  concerning  the  content  and 
interpretation  of  this  guide  should  be 
addressed  to  JXXX/JFS,  XX  Road,  Any 
Town,  USA  XXXXX-XXXX. 

4 .  CLASSIFICATION 
RECOMMENDATIONS.  Recommended 
changes  will  be  completely  documented  and 
justified.  Recommendations  should  be  made 
through  appropriate  channels  to  the  OPR. 
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Pending  final  decision,  contested  classified 
information  shall  be  handled  and  protected  at 
the  higher  classification.  All  users  are 
encouraged  to  assist  in  improving  and 
maintaining  the  currency  and  adequacy  of  this 
guide. 

Because  of  interface  relationships  of  program 
material,  unclassified  items  of  information 
may  become  classified  or  sensitive  when 
combined  into  a  single  document.  Therefore, 
care  must  be  taken  prior  to  the  release  of  any 
related  information,  even  though  it  is 
unclassified.  Further,  the  release  of  such 
information  should  be  based  upon  the 
recipient's  strict  need-to-know. 

5 .  APPLICATION,  REPRODUCTION, 
AND  DISSEMINATION.  This  guide  is 
intended  specifically  for  the  use  and  guidance 
of  individuals  with  a  need-to-know 


requirement  for  information  contained  herein. 
Reproduction  and  dissemination  will  be  at  the 
discretion  of  the  FSD. 

6 .  PUBLIC  RELEASE.  The  fact  that  certain 
information  is  shown  to  be  unclassified  does 
not  authorize  public  release.  Public  release  of 
unclassified  information  must  be  approved  by 
the  FSD. 

7 .  FOREIGN  DISCLOSURE.  Release  of 
program  information  or  hardware/software  is 
not  authorized  to  any  foreign  entity  or  foreign 
national  without  prior  approval  by  the 
JXXX/FSD.  All  proposed  releases  of 
program  information  to  any  foreign 
government  shall  be  forwarded  to  the  OPR 
for  review  and  further  processing  action. 

8.  PROGRAM  DEFINITIONS.  The 

following  is  a  list  of  abbreviations  and 
designations  used  throughout  this  guide: 


a.  C 

Confidential  Classification  Level 

b.  COMSEC 

Communications  Security 

c.  DoD 

Department  of  Defense 

d.  FOUO 

For  Official  Use  Only 

e.  FSD 

Feasibility  Study  Director 

f.  JXXX 

Joint  XXX  Feasibility  Study 

g.  JFS 

Joint  Feasibility  Study 

h.  JTD 

Joint  Test  Director 

i.  JTF 

Joint  Test  Force 

j.  JT&E 

Joint  Test  &  Evaluation 

k.  MACOM 

Major  Command 

1.  NOFORM 

Not  Releasable  to  Foreign  Nationals 

m.  OADR 

Originating  Agency  Determination  Required 
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n.  OPR 

Office  of  Primary  Responsibility 

o.  OPSEC 

Operations  Security 

p.  R&D 

Research  and  Development 

q.  RCS 

Radar  Cross  Section 

r.  RDT&E 

Research,  Development,  Test  &  Evaluation 

s.  S 

Secret  Classification  Level 

t.  SAM 

Surface-to-Air  Missiles 

u.  SCG 

Security  Classification  Guide 

v.  T&E 

Test  and  Evaluation 

w.  TS 

Top  Secret  Classification  Level 

x.  TSPI 

Time  Space  Positioning  Information 

y.  U 

Unclassified 

z.  WNINTEL 

Warning  Sensitive  Intelligence  Sources  and  Methods  Involved 

C.  SECTION  II 

OVERALL  EFFORT 

9.  GOALS,  MISSION,  &  PURPOSE.  To 

provide  instructions  and  guidance  for  the 
security  classification  of  information  and 
material  originated  by  the  JXXX/JFS. 

a.  The  purpose  of  the  JXXX/JFS  is  to 
determine  the  need  for  and  feasibility  of  a 
joint  test  that  is  to  (as  appropriate) 

b.  The  JXXX/JFS  Security  Classification 
Guide  (SCG)  is  designed  to  protect 
information  related  to  (  e.g.,  U.S. 
missiles,  manned  aircraft  systems,  sensor 
performance  and  vulnerabilities, 
intelligence  assets  and  All-Source 
Imagery  information,  tactics  and  weapons 
employment,  CCD  treatments,  etc.),  data 


collected  or  derived  from  other  sources 
for  JXXX  use  and  various  other  aspects 
of  the  test  procedures. 

10.  END  ITEM.  The  end  item(s)  of  this 
effort  include  a  Joint  Feasibility  Study  Final 
Report  and  an  Analysis  Plan  for 
Assessment. 

The  JFS  final  report  and  APA  will  normally 
be  unclassified  unless  the  subject  matter  is 
otherwise  classified  or  so  designated  by  the 
authority  or  agency  supplying  the  proposed 
equipment  and/or  techniques  to  be  tested. 
Test  data  and  results  which  indicate  a 
capability,  vulnerability,  weaknesses  or 
which  would  compromise  system 
survivability  will  normally  be  classified 
Secret.  Actual  use  of  equipment  or 
concepts  to  be  tested  during  exercises, 
alerts,  and  war  will  be  classified  in 
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accordance  with  the  guidance  of  the  theater, 
area  of  operations,  or  base  commanders. 


which  is  classified  by  other  agencies  are 
utilized.  The  following  subparagraphs 
contain  detailed  classification  guidance. 

Source  classification  guidance  must  be 
consulted  when  information  and  hardware 


TOPIC  CLASS  DECLASS  COMMENT 

11,  GENERAL: 

a.  Information  revealing  the  U 

existence  of  general  (XXX) 
policy,  doctrine,  tactics, 
material,  techniques,  and 
training  program 

b.  Acknowledgment  that  new  U 

(XXX)  capabilities  are 
continually  being  developed 

c.  The  identity  of  U  Protect  as  FOUO 

organizational  (XXX) 

planners 

d.  Information  that  reveals  that  U  Note  2 

there  is  a  policy  of 

employing  peacetime 
(XXX)  to  support  security 
operations 

e.  Information  that  reveals  that  U  Protect  as  FOUO 

there  is  a  policy  of 

employing  (XXX)  to 
enhance  defense  of  forces 
and/or  support/conduct 
combat  operations 

f .  Information  that  reveals  a  U  Notes  2  &  3 

policy  concerning  the 

planning  and  conduct  of 
combined  (XXX)  operations 
and  exercises 
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g.  Generic  tactics  and  U 

techniques  such  as  feints, 
diversions,  ruses, 
demonstrations,  displays, 
electronic  deceptions,  and 
communications 

h.  Effectiveness  of  actual  U 

tactics  or  techniques,  if 

known,  that  are  not 
influenced  by  foreign 
knowledge 

i.  Actual  tactics  or  techniques  S 

which,  if  known,  would  aid 

an  adversary  in  the  detection 
of  tactical  (XXX)  activity  or 
facilitate  development  of 
countermeasures 

j.  Actual  tactics  or  techniques  See 

which,  if  disclosed,  might 

reveal  intelligence  sources  or  Comments 
collection  capabilities 

k.  Information  detailing  the  S 

comprehensive  capabilities 

and  goals  (current  and 
projected)  of  a  service, 

MACOM,  base  or  unit 

l.  Information  identifying  S 

vulnerabilities  or 

deficiencies  in  tactical 
(XXX)  operational  plans, 
conceptual  plans, 
operations,  capabilities, 
tactics,  training,  or 
equipment  which  would 
negate  their  effect  if  known 


OADR 


OADR 


OADR 


Note  3 


Notes  4  &  9 


Note  3 
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m.  Information  revealing  the  S  OADR  Note  3 

actual  employment  of 
(XXX)  equipment  and 
techniques  when  knowledge 
of  the  information  would  aid 
threat  planning  to  negate, 
nullify,  or  counter  the  use  of 
the  means;  or,  when  the 
existence  of  a  classified 
capability  would  be  revealed 

12.  (XXX)  EQUIPMENT 
AND  TECHNIQUES 

a.  Permanent  (XXX)  U  Note  2 

equipment/techniques  used 

and/or  employed 

b.  Plans  for  specific  use  of  S  OADR 

(XXX)  which  are  deployed 

only  during  certain  stages  of 
alert  or  actual  combat 

c.  Instructions  revealing  how  U 

to  use  (XXX)  equipment 

and  techniques  (unclassified 
unless  otherwise  noted) 

13.  TEST  DATA: 

a.  Test  plans,  methodology,  U 

test  data,  or  results  that  do 

not  reveal  specific 
vulnerabilities,  capabilities, 
or  weaknesses 

b.  Test  plans,  methodology,  S  OADR 

test  data,  or  results  that 

reveal  specific 
vulnerabilities,  capabilities, 
or  weaknesses 
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c.  Test  philosophy  & 
objectives 


•  If  capabilities, 
vulnerabilities  or 

limitations  are  revealed 

S 

OADR 

•  If  capabilities, 
vulnerabilities,  or 
limitations  are  not 
revealed 

U 

d.  Test  results 

•  If  capabilities, 
vulnerabilities,  or 
limitations  are  revealed 

s 

OADR 

•  If  capabilities, 
vulnerabilities,  or 
limitations  are  not 
revealed 

u 

e.  Actual  test  dates 

u 

Protect  as  FOUO 

f .  Overall  test  schedule 

u 

Protect  as  FOUO 

g.  Location  of  test  sites 

u 

14.  DATA  ACQUISITION, 
REDUCTION,  AND 
PRESENTATION: 

a.  Mathematical  algorithms  or 
computer  software  routines 
that  are  used  to  reduce, 
analyze,  or  present  data 

u 

Note  6  (Information  may  be 
protected  as  proprietary) 

b .  Mathematical  algorithms  or 
computer  software  routines 

u 

Note  6  (Information  may  be 
protected  as  proprietary) 

that  are  used  to  evaluate 
(XXX)  equipment  or 
techniques  effectiveness 
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c.  Remotely  sensed  signatures,  S  OADR  Notes  7  &  8 

measurements,  reports, 

papers,  software,  synthetic 
or  enhanced  imagery, 
drawings,  etc.,  that  provide 
research  and  development 
results,  conclusions,  or 
recommendations  that  reveal 
an  enhancement  or 
vulnerability  that  could 
compromise  the 
survivability  or  operability 
of  a  high-value  asset  target, 
facility  or  installation 

d.  Specific  conclusions  and  S  OADR 

recommendations  regarding 

(XXX)  test  and  evaluation 
drawn  from  raw  or  tabulated 
data  as  described  in  (c)  and 
processed  data  as  previously 
described  if  they  reveal 
vulnerabilities  or  impact  on 
survivability 

e.  Reports  resulting  from  use  See  Note  11 

of  classified  data  and/or  Comments 

analysis,  conclusions,  and 
recommendations 

f .  Compiled  database  (in  any  S  OADR  Notes  6  &  8 

form)  of  test  results  when 

capabilities,  vulnerabilities, 
limitations,  or  weaknesses 
are  revealed 

g.  Compiled  database  (in  any  See  Note  8  (Protect  as  FOUO) 

form)  of  test  results  when  Comments 

capabilities,  vulnerabilities, 
limitations,  or  weaknesses 
are  not  revealed 
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D.  SECTION  III 


PERFORMANCE  AND  CAPABILITIES 
TOPIC  CLASS  DECLASS 

15.  TEST  &  EVALUATION 

a.  Information  related  to  the  S  OADR 

performance  and/or 

capabilities  of  specific 
(XXX)  equipment  or 
techniques 

b.  Information  related  to  S  OADR 

countermeasures  that  would 

affect  the  performance 
and/or  capabilities  of  (XXX) 
equipment  or  techniques 

c.  Information  regarding  S  OADR 

current  or  projected 

performance  and/or 
capabilities  of  (XXX) 
equipment  or  techniques 

d.  Design,  performance  and/or  See 

functional  characteristics  of  Comments 

operational  (XXX) 

equipment 

e.  Test  procedures  information 

•  If  operational  S  OADR 

employment  tactics, 

capabilities,  or 
limitations  are  revealed 

•  If  operational  U 

employment  tactics, 
capabilities,  or 

limitations  are  not 
revealed 


COMMENT 


Note  3 


Note  9 


Note  9 
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16.  DATA  REDUCTION  & 
PRESENTATION: 


a.  Test  data  and  reports  that  See 

disclose  (XXX)  Comments 

performance 

b .  Test  data  and  reports  that  do  U 
not  disclose  (XXX) 
performance 

c.  Test  data  analysis,  results, 
and  conclusions 

•  If  capabilities,  S  OADR 

limitations,  or 

vulnerabilities  of  a 
system  are  revealed 

•  If  capabilities,  U 

limitations,  or 
vulnerabilities  of  a 

system  are  not  revealed 

d.  Flight  test  data/encryption  See 

codes  Comments 

e.  Raw  or  tabulated  data  U 

resulting  from  tests,  such  as 
uncaptioned  photographs, 
thermographs,  temperature 
measurements,  radar  cross- 
section  measurements,  etc. 

f .  Processed  data  from  tests,  S  OADR 

such  as  captioned 

photographs  or 
thermographs,  tabulated 
temperature  measurements, 

RCS  measurements,  etc. 

g.  Mission  tapes  obtained  from  See 

test  aircraft  Comments 


Note  9 


Note  3 


Notes  4  &  9 


Notes  7  &  9 
(Protect  as  FOUO) 


Notes  4  &  10 


Note  9 

(See  Sec.  20  a  &  b) 
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h .  Raw  test  data  tapes  and  logs 
generated  by  JXXX 

•  If  results  are  revealed  S  OADR 

which  indicate 

capabilities, 
vulnerabilities,  or 
weaknesses 

•  If  results  are  not  U 

revealed  which  indicate 
capabilities, 
vulnerabilities,  or 
weaknesses 

i.  Measurement  parameters  See 

identified  on  data  sets,  Comments 

imagery  and  individual 

target  RCS  tables 

j .  Display  of  mission  route  or  U 
maps  containing  information 
revealing  location,  scale  of 
distance 

17.  (XXX)  DESIGN 
INFORMATION: 

a.  If  information  concerning  S  OADR 

limitations,  deficiencies, 

vulnerabilities  or 
weaknesses  that  could 
impact  on  system 
survivability  is  revealed 

b .  If  information  concerning  U 

limitations,  deficiencies, 
vulnerabilities  or 

weaknesses  that  could 
impact  on  system 
survivability  is  not  revealed 


Notes  4  &  9 


Notes  3  &  9 
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E.  SECTION  IV 


SPECIFICATIONS 

TOPIC  CLASS  DECLASS 

18.  SYSTEM  AND  SUB¬ 
SYSTEM  SPECIFICA¬ 
TIONS: 

a .  T&E  activities  and  end 
products  that  reveal: 

•  Specifications  of  (XXX)  U 
equipment  or  techniques 

that  describe  items  but 
do  not  provide 
information  regarding 
signature  alteration, 
radar  cross-sections, 
etc/as  appropriate 

•  Specifications  of  (XXX)  S  OADR 

equipment  or  techniques 

that  describe  items  and 
provide  information 
regarding  signature 
alteration,  radar  cross- 
sections,  etc./as 
appropriate 

F.  SECTION  V 

CRITICAL  ELEMENTS 
TOPIC  CLASS  DECLASS 

19.  TEST  &  EVALUATION 

a.  Activities  which  reveal  test 


COMMENT 


Note  9 


COMMENT 
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•  The  fact  that  equipment 
and  techniques  have  the 
capability  to  alter  or 
simulate  signatures  in 
any  spectral  band 

S 

OADR 

•  Those  features  of 
equipment  and 
techniques  that  provide 
levels  of  signature 
alteration  or  simulation 

S 

OADR 

b .  The  fact  that  equipment  and 
techniques  cause 
degradation  in  enemy  target 
detection  and  recognition 
capability 

U 

c.  Quantitative  data  on  actual 
degradation  in  detection  and 
recognition  range  of  target 
acquisition  systems 

s 

OADR 

d .  Conclusions  regarding  the 
current  and/or  projected 
impact  on  target  detection 
and  recognition  capability 

s 

OADR 

e.  New  principles,  procedures, 
or  doctrine  that  reveal  a 
break-through  with 
significant  military  impact 

s 

OADR 

f .  Operational  concepts  for 
implementation  of  CCD 
equipment  and  techniques 

s 

OADR 

.  DATA  ACQUISITION: 

a.  Time  Space  Positioning 
Information  (TSPI) 

See 

Comments 

Note 

b .  HUD  systems  data 

See 

Comments 

Note 
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21.  AIRBORNE  SENSORS 
AND  INTELLIGENCE 
ASSETS: 

G .  SECTION  VI 

VULNERABILITIES  AND  WEAKNESSES 
TOPIC  CLASS  DECLASS  COMMENT 

22.  (XXX)  EQUIPMENT  & 

TECHNIQUES: 

a.  Instructions  on  how  to  ( as  S  OADR 

appropriate,  i.e.  operate, 

assemble,  erect,  move,  or 
place)  (XXX)  equipment  or 
when  a  vulnerability  or 
weakness  is  indicated 

b.  Current  or  projected  (XXX)  S  OADR 

equipment  and  techniques 

that  reveal  vulnerabilities  or 
weaknesses 

c.  Information  resulting  from  S  OADR  Note  3 

tests  regarding  current  or 

projected  (XXX)  equipment 
or  techniques,  deployment 
concepts,  or  counter¬ 
countermeasures  that  reveal 
an  enhancement  or 
vulnerability 

23.  DATA  REDUCTION: 

a.  Mathematical  techniques  and 
computer  codes  that  are 
related  to  assessment  of 
(XXX)  vulnerabilities  and 
weaknesses: 
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•  Aid  in  the  solution,  but 
do  not  predict  signatures 
from  physical  data 
inputs 

U 

Note  8 

•  Predict  signatures  from 
physical  data  inputs  if 
vulnerabilities  are 
revealed 

S 

OADR 

Note  9 

•  Include  optimization 
routines  for  defining 
configuration  shapes  for 
low  or  minimal 
signatures 

S 

OADR 

.  SUPPORT  EQUIPMENT: 

a.  Aircraft 

See 

Comments 

Note  9 

b.  Ground  equipment 

See 

Comments 

Note  9 

H.  SECTION  VII 


TOPIC 
25.  HARDWARE: 

a.  Hardware  nomenclature 

b .  Federal  stock  number 

c.  Serial  numbers 

d.  Model  designation 

e.  (XXX)  End  item(s)  (as 
appropriate) 

f .  Support  aircraft  (as 
appropriate) 


HARDWARE 
CLASS  DECLASS 

U 

u 

u 

u 

See 

Comments 

See 

Comments 


COMMENT 


Note  5 

Notes  5  &  9 
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g.  Ground  support  equipment 

See 

Notes  5  &  9 

(as  appropriate) 

Comments 

26.  DATA  PROCESSING 

EQUIPMENT: 

a.  Specification/performance 

See 

Notes  1,5  All 

characteristics  equipment  or 
instrumentation  used  to 
collect,  reduce,  analyze,  or 
present  data  or  used  to 
evaluate  (XXX) 

Comments 

(Information  may  be 
supplier  proprietary) 

I.  SECTION  VIII 


ADMINISTRATIVE  DATA 


TOPIC  CLASS  DECLASS 


27.  GENERAL: 

a.  (XXX)  equipment  and  U 

techniques  except  for  those 
hardware  and  data  items 
including  signature 
measurements, 

photographs,  drawings,  and 
specifications 

b.  Specialized  (XXX)  S 

equipment  and  techniques 

for  critical  elements 

c.  Orders  to  install,  erect,  or 
place  individual  items  of 
(XXX)  equipment 

•  Pertaining  to  wartime  S 

measures  that  would 
indicate  preparations 
against  or  for  attack 


OADR 


OADR 


COMMENT 


Note  5 
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•  If  items  do  not  pertain  to  U 
wartime  measures  that 
would  indicate 
preparations  against  or 
for  attack 

d.  Names  and  locations  of  U 

(XXX)  test  program 
participants 

28.  DEGREE  OF 
PROTECTION  IN 
TRANSIT/STORAGE/ 

PACKAGING: 

a.  When  the  information  S  OADR 

reveals  an  operational 

deficiency/limitation 

b .  When  the  information  S  OADR 

reveals  a  classified  plan  or 

operation 

c.  When  the  information  does  U 

not  apply  to  (a)  or  (b) 

29.  T&E  PROGRAM: 

a.  Funding  (individual  U 

contracts  or  projects, 

program,  total  T&E  effort) 

b .  Budget  year,  prior  year(s),  U 

future  fiscal  year(s)  or  total 
dollars 

c.  Quantities  by  budget  or  U 

future  fiscal  year(s) 

30.  PRODUCTION  AND 
PROCUREMENT: 

a.  Contract  delivery  schedules  U 

of  (XXX)  equipment  and 
material 


Protect  as  FOUO 


Note  12 


Note  12 


Protect  as  FOUO 
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b.  Delivery  schedules  for  S  OADR 

equipment  and  material 
needed  in  conduct  of  a 
test/project  when 
information  reveals 
operational 

deficiency/limitation  or 
anticipated  fielding  plan  for 
urgently  needed  or  new 
technology  system(s) 


J.  SECTION  IX 


NOTES 


1 .  Security  classification  must  be  determined 
by  detailed  considerations  of  all  applicable 
elements.  When  partial  information  is 
presented,  the  lowest  classification 
consistent  with  adequate  protection  will  be 
assigned. 

2.  The  establishment  of  security  classification 
levels  for  the  deployment  data  within  a 
theater  is  the  responsibility  of  the  Theater 
Commander. 

3.  Any  information  revealing  possible 
countermeasures  and  counter- 
countermeasures  to  U.S.  military  (XXX) 
equipment/systems  will  normally  be 
classified  SECRET.  Any  detailed 
discussion,  report,  or  study  of  system 
capabilities  and/or  limitations  from  which 
conclusions  may  be  drawn  that  would 
suggest  enemy  tactics  most  likely  to  defeat 
U.S.  equipment/systems  and  techniques 
will  be  classified  SECRET. 


4.  Classify  in  accordance  with  source  material 
or  information  revealed. 

5.  Equipment  shall  be  safeguarded  in 
accordance  with  the  requirements  for  the 
category  of  information  revealed  by  the 
equipment.  Commercial  equipment  is 
normally  unclassified.  Specifically 
designed  equipment  and/or  specifically 
designed  equipment  utilizing  commercial 
equipment  will  be  assigned  a  security 
classification  commensurate  with  the 
performance  characteristics  or  parameters 
of  information  that  is  revealed. 

6.  Security  classification  of  computer 
programs,  or  segments  thereof,  shall  be 
based  on  the  classification  of  the 
information  or  parameters  that  can  be 
revealed  by  the  program.  In  order  to 
accurately  classify  a  particular  program, 
review  applicable  SCGs  and  apply  only  that 
level  of  classification  required  to  protect  the 
parameters  contained  in  the  program.  This 
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note  applies  to  interim  and  final  products 
(tape,  card  deck,  listings,  etc)  as  well  as 
any  software  documentation  which  can 
reveal  the  classified  data. 

7.  Reports,  publications,  drawings, 
schematics,  photographs,  models, 
mockups,  training  aids,  and  graphs  will  be 
assigned  a  security  classification  as 
determined  by  the  appropriate  application  of 
this  security  classification  guide  and 
information  revealed. 

8 .  Unclassified  items  may  become  sensitive  or 
classified  when  compiled.  See  individual 
sections  for  overall  highest  classification. 

9.  Consult  appropriate  classification 
guidelines  which  are  applicable  to  unique 
U.S.  Military  aircraft,  ground  support 


equipment  and  systems,  collection  systems, 
facilities,  computer  programs,  and 
operational  tactics. 

10.  The  relationship  of  data  to  telemetry 
channel,  calibration,  raw  telemetry,  and 
frequency  of  operation  shall  be  classified  in 
accordance  with  the  information  which 
might  be  discernible  from  them.  The 
telemetry  encryption  shall  be  SECRET  and 
raw  telemetry  shall  be  unclassified.  The 
relationship  of  raw  telemetry  to  calibration 
is  classified  by  the  data  revealed. 

11.  Classify  end  item  in  accordance  with  the 
highest  classified  item  revealed. 

12.  Movement  and  storage  of  classified 
hardware  between  installation  and  activities 
will  be  accomplished  in  accordance  with  the 
provisions  of  DoD  5200. 1R  and  DoD 
5220.22M. 
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ANNEX  M 

MILESTONE  REVIEWS 


A.  INTRODUCTION 

DDT&E  will  normally  review  the  progress  of  the 
JFS  at  three  established  milestones:  when  the 
program  issues  are  solidified,  upon  completion 
of  the  program  test  concept,  and  after 
coordination  of  the  JFS  final  report.  This  annex 
provides  guidance  for  these  reviews  in  the  form 
of  suggested  outlines  for  the  review 
presentations.  These  suggested  outlines  are 
provided  as  formats  that  should  be  tailored  to  the 
specific  requirements  of  the  JFS  concepts  and/or 
issues. 

The  first  milestone  review  should  be  an 
IPR  that  will  be  conducted  for  DDT&E  after  the 


JFS  is  organized  and  the  concepts  and/or  issues 
have  been  solidified.  The  second  milestone 
should  be  a  TAB  review  of  the  JT&E  concept. 
The  focus  of  this  review  will  be  on  the  technical 
aspects  of  the  JT&E  concept  in  terms  of 
credibility  and  risks.  The  final  milestone  should 
be  a  TAB/DDT&E  review  of  the  JFS  final  report 
and  the  APA.  This  review  will  be  conducted 
after  both  documents  have  been  coordinated  with 
all  interested  agencies.  The  focus  of  this  review 
will  be  the  results  of  the  JFS  and,  if  a  favorable 
chartering  decision  is  anticipated,  the  technical 
competency  and  credibility  of  the  APA. 
Suggested  outlines  for  these  reviews  are 
contained  in  the  following  sections. 


B.  SUGGESTED  OUTLINE  FOR  FIRST  MILESTONE  REVIEW  (IPR) 

1 . 0  Introduction 

1.1  Title/Nomination  Organization/Agency 

1.2  Lead  Service/Executive  Agency/FSD 

1 .3  Supporting  Services/Executive  Agencies 

1 . 4  Other  Maj or  Players 

1 .5  Purpose  and  Focus  of  Review  (establishment  of  JFS  and  solidification  of  concepts  and/or 
issues) 

2.0  Background 

2.1  Nomination  and  Problem  Statement 

2.1.1  Limitations 

2.1.2  Constraints 

2.2  Feasibility  Study 

2.2.1  Purpose 

2.2.2  Test  Concepts  and/or  Issues  and  Objectives 

2.2.3  Test  Articles 

2.3  Scope  of  Study 

2.3.1  Mission  Requirements 

2.3.2  System  Requirements 
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2.3.3  Participants 

2.3.4  Users  of  JT&E  Test  Results 

2.3.5  Focus  of  JT&E 

3 . 0  JFS  Establishment 

3.1  Facilities 

3.2  Staff 

3.2.1  Identify/Quantify 

3.2.2  Formal  Request  to  Service  POC 

3.2.3  Organization/Responsibilities 

3 . 3  Communications  and  ADP 

3.4  Funds 

3.4.1  Original  Budget  Estimate 

3.4.2  Revised  Budget  Estimate 

3.5  JFS  Schedule 

3.6  Security 

3.6.1  Security  Classification  Guide 

3.6.2  Procedure 

3.6.3  SCIF  Requirements 

3 . 7  Coordination  Chain 

3.7.1  Signature  Authority 

4.0  Concepts/Issues 

4. 1  JT&E  Problem  Statement  (as  refined  by  FSD) 

4.2  Concepts/Issues  Justification  Statement 

4.3  Coordination 

4.3.1  Organizations 

4.3.2  Recommended  Revisions 

4.3.3  Rectification  of  Recommendations 

4.4  Solidified  Issues 

5.0  Problems 

6.0  Status  Summary  of  JFS  and  Recommendations 

C.  SUGGESTED  OUTLINE  FOR  SECOND  MILESTONE  REVIEW  (JT&E 
CONCEPT) 

1.0  Introduction 

1.1  Purpose  of  JT&E 

1.2  Program  Overview/Direction 

1.2.1  Background 

1.2.2  Authority 
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1.3  JT&E  Focus 

1.4  Environment 

1.5  Is  the  JT&E  Necessary? 

1.6  Summary 

2.0  Joint  Test  &  Evaluation  Concept  Description 

2. 1  Nomination  Concerns  and  Problems 

2.2  Goals  and  Objectives 

2.3  Limitations  and  Constraints 

2.4  Scope  of  JT&E  Concept 

2.4.1  Mission  Requirements 

2.4.2  System  Requirements 

2.5  Issues 

2.6  Test  Articles 

2.7  Models  and  Simulations 

2.8  Overall 

3.0  JT&E  Test  Methodology 

3 . 1  Structured  Breakdown  of  Issues 

3.1.1  Effectiveness  Issues 

•  Measures 

•  Data  Elements 

3.1.2  Suitability  Issues 

•  Measures 

•  Data  Elements 

3.2  Data  Collection  Methodology 

3.2.1  Data  Collection 

•  Data  Elements 

•  Controlled/Uncontrolled  Variables 

3.2.2  Data  Processing 

3.2.3  Data  Storage  and  Retrieval 

3.3  Data  Analysis  Methodology 

3.3.1  Statistical  Significance 

3.3.2  Credibility  of  Analysis  Results 

4.0  Joint  Test  &  Evaluation  Concept  Approach 

4.1  Test  Scenario 

4.2  Approach  for  Resolving  Issues 

4.2.1  Pretest/T  est/Posttest 

4.2.2  Test  Methods 

4.2.3  Test  Cells 

4.2.4  Test  Opportunities 

4.2.5  Test  Activity  Schedule 

4.3  JT&E  Products 
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5.0  Program  Management 


D .  SUGGESTED  OUTLINE  FOR  THIRD  MILESTONE  REVIEW  (JFS  FINAL 
REPORT) 

Executive  Summary 
1 . 0  Introduction 

2.0  Joint  Feasibility  Study  Results 

2.1  Literature  Search 

2.2  Study  Results 

2.2.1  Determination  of  User  Requirements 

2.2.2  Test  Objectives  based  on  User  Requirements 

2.2.3  Relationship  to  Other  Activities 

2.2.4  Determination  of  JT &E  Need 

2.2.5  Determination  of  JT &E  Feasibility 

2.2.6  Analysis  Plan  for  Assessment 

2 . 3  Other  Accomplishments 

2.4  JFS  Products 

2.5  Recommended  Structure  and  Location  for  JTF  if  Chartered 

3.0  Problems  and  Solutions 

3 . 1  Urgency  of  Need  for  Test  Results 

3 . 2  Staffing  Lead  Times 

3.3  Systemic  Problems  and  Lessons  Learned 

3.4  Unique  or  new  Problems  and  Lessons  Learned 

4.0  Conclusions  and  Recommendations 

4.1  OSD,  Joint  Staff  and  Service  Concurrence 

4.2  Expected  Products 

4 . 3  Benefit  of  Participation 

4.4  Conclusions 

4.5  Recommendations 


Annexes  (as  required  to  substantiate  conclusions  and  recommendations) 
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E.  ANALYSIS  PLAN  FOR  ASSESSMENT  (APA)  (if  the  JFS  conclusion  and 
recommendation  are  to  charter  a  JTF  to  conduct  a  JT&E,  the  third  milestone 
should  include  a  review  of  the  APA) 

Executive  Summary 

1 . 0  Introduction 

1 . 1  Purpose  of  the  JT &E 

1.2  JT&E  Overview/Direction 

1.2.1  Background 

1.2.2  Authority 

1.3  JT&E  Focus 

1.4  Environment 

1.5  Summary 

2.0  Description  of  JT&E 

2.1  Nomination  Concerns  and  Problems 

2.2  JT &E  Obj  ectives 

2.3  Limitations  and  Constraints 

2.4  Scope  of  JT&E 

2.4.1  Mission  Requirements 

2.4.2  System  Requirements 

2.5  JT&E  Concepts  and/or  Issues 

2.6  JT &E  T  est  Articles 

2.7  JT &E  Environments 

3 . 0  JT&E  Test  Methodology  (focus  on  any  deviations  from  the  JT&E  concept) 

3.1  Structured  Breakdown  of  Issues 

3.1.1  Effectiveness  Issues 

•  Measures 

•  Data  Elements 

3.1.2  Suitability  Issues 

•  Measures 

•  Data  Elements 

3 . 2  Data  Collection  Process 

3.2.1  Data  Collection 

•  Data  Elements 

•  Controlled/Uncontrolled  Variables 

3.2.2  Data  Processing 

3.2.3  Data  Storage  and  Retrieval 

3.3  Data  Analysis  Process 

3.3.1  Statistical  Significance 

3.3.2  Credibility  of  Analysis  Results 
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4 . 0  JT&E  Test  Approach  (focus  is  on  any  deviations  from  the  JT&E  concept) 

4.1  Scenario 

4.2  Approaches  for  Resolving  Issues 

4.2.1  Pretest/Test/Posttest 

4.2.2  Test  Methods 

4.2.3  Test  Cells 

4.2.4  Test  Opportunities 

4.2.5  Test  Activities  Schedule 

4.3  JT&E  Products 


5.0  Data  Management 

5 . 1  Data  Management  Systems 

5.2  Hardware  and  Software  Systems 

5 . 3  Data  Handling  Process 

5.4  Data  Validation  Process 

5 .5  Data  Analysis  and  Evaluation  Process 

5 . 6  Analysis  Reporting 

6 . 0  Program  Management 

Annexes  (as  required  to  substantiate  the  credibility  of  the  APA) 
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ANNEX  N 

SUGGESTED  OUTLINES  FOR  THE  JFS  FINAL  REPORT, 
MANAGEMENT  REPORT,  AND  ANALYSIS  PLAN  FOR  ASSESSMENT 

A.  INTRODUCTION 

The  suggested  outlines  should  be  tailored  to  meet  the  specific  requirements  of  each  joint 
feasibility  study. 

B.  SUGGESTED  OUTLINE  FOR  THE  JOINT  FEASIBILITY  STUDY  FINAL 
REPORT 

Executive  Summary 

1.0  Introduction/Background 

1 . 1  Nomination  and  Problem  Statement 

1.1.1  Assumptions 

1.1.2  Limitations  and  Constraints 

1.2  Feasibility  Study  Purpose 

1.2.1  Test  &  Evaluation  Concepts  or  Issues  and  Objectives 

1.2.2  Test  Articles 

1.3  Scope  of  Study 

1.3.1  Mission  Requirements 

1.3.2  System  Requirements 

1.3.3  Participants  and  Users  of  Test  &  Evaluation  Results 

1.3.4  T est  &  Evaluation  Focus 

2.0  Task  Needs 

2.1  Test  Requirements 

2.2  Scenarios 

2.3  Decomposition  of  Concepts  or  Issues 

2.3.1  MLMs 

2.3.2  MOEs 

2.3.3  MOPs 

2 . 4  The  Aggregation  of  Measures 

2.5  Data  Requirements 

2.5.1  Baseline 

2.5.2  Types  of  Data  Required 

2.6  Test  Resource  Requirements 

2.6.1  Proposed  Exercise  Participation/Test  Sites 

2.6.2  Schedule 
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2.6.3  Funding 

2.6.4  Project  Management 

2.6.5  Personnel  Support 

2.6.6  Equipment  and  Instrumentation 

3.0  Joint  Feasibility  Study  Methodology 

3.1  Ground  Rules 

3.1.1  Directions 

3.1.2  Assumptions 

3 . 2  Alternative  T  est  Methods 

3.2.1  Minimum  Size  Test  Program 

3.2.2  Optimum  Size  Test  Program 

3 .3  Summary  of  Test  Concept 

3.3.1  Data  Collection 

3.3.2  Analysis  Plan 

4.0  Joint  Feasibility  Study  Results 

4 . 1  Literature  Search 

4.2  Study  Results 

4.2.1  Determination  of  User  Requirements 

4.2.2  Test  &  Evaluation  Objectives  Based  on  User  Requirements 

4.2.3  Relationship  to  Other  Activities 

4.2.4  Determination  of  JT&E  Need 

4.2.5  Analysis  Plan  for  Assessment 

4.2.6  Determination  of  JT &E  Feasibility 

4.3  Other  Accomplishments 

4.4  JFS  Products 

5.0  Problems  and  Solutions 

5 . 1  Staffing  Lead  Time 

5.2  Others 

6.0  Conclusions  and  Recommendations 

6.1  OSD,  Joint  Staff  and  Service  Coordination  /Concurrence 

6.2  Expected  Products 

6 . 3  Benefits  of  Participation 

6.4  Conclusions 

6.5  Recommendations 

6.6  Recommendations  Relative  to  JTF  (if  charter  is  recommended) 

6.6.1  Location 

6.6.2  Staffing 
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6.6.3  Estimate  of  Funding  Requirements  by  Fiscal  Year  and  Categorized  by  Each  Service  and 
OSD 

6.6.4  Estimate  of  Resources  Requirements  by  Fiscal  Year  and  Categorized  by  Each  Service 
and  OSD 

Annexes  (as  required  to  substantiate  conclusions  and  recommendations) 

C.  SUGGESTED  OUTLINE  FOR  THE  JOINT  FEASIBILITY  STUDY 
MANAGEMENT  REPORT  (not  required  if  recommending  that  JT&E  be 
chartered) 

Executive  Summary 

1 . 0  Introduction 

1 . 1  Purpose  of  Report 

2.0  Assessment  of  Accomplishments 

2. 1  Relative  to  First  Chartered  Objective 

2.2  Relative  to  Second  Chartered  Objective 

2.3  Relative  to  Third  Chartered  Objective 

2 .n  Relative  to  n  Chartered  Objective 

3 . 0  Lessons  Learned 

3.1  Same  or  Similar  to  Lessons  Learned  That  Were  documented  by  Previous  JFSs  or  JTFs 

3.1.1  Lesson  #1 

3.1.2  Lesson  #2 

3.1.3  Lesson  #3 

3.1. n  Lesson  #n 

3.2  Lessons  Learned  That  Were  Unique  to  this  Joint  Feasibility  Study 

3.2.1  Lesson  #1 

3.2.2  Lesson  #2 

3.2.3  Lesson  #3 

3.2. n  Lesson  #n 

4.0  Recommendations 

4.1  Recommendations  (1-n) 

D .  SUGGESTED  OUTLINE  FOR  THE  ANALYSIS  PLAN  FOR  ASSESSMENT 
Executive  Summary 

1 . 0  Introduction 

1.1  Purpose/Authority 
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1.2 

1.3 


Background/Problem 
Joint  Feasibility  Study  Results 

1.3.1  Is  a  JT&E  Needed? 

1.3.2  Is  a  JT&E  Feasible? 

2 . 0  JT&E  Description 

2.1  Concepts,  Issues,  and  Objectives 

2.1.1  Coordination 

2.1.2  Approval 

2.2  Scope  of  JT&E 

2.2.1  Test  Articles/Methodologies/Participants 

2.2.2  Assumptions 

2.2.3  Constraints  and  Limitations 

2.3  Test  Scenario 

2.4  Test  Measures 

2.4.1  Mission  level  measures 

2 . 4 . 1 . 1  Data  elements 

2.4.2  Effectiveness  Measures 

2 . 4 . 2 . 1  Data  Elements 

2.4.3  Suitability  Measures 

2 . 4 . 3 . 1  Data  Elements 

2.5  Test  Concept 

2.5.1  Approach  for  Concepts/Issue  Resolution 

2.5.2  Test  Methods 

2.5.2.1  Cost/Credibility  of  Results 

2. 5. 2. 2  Availability/Technical  Maturity 

2.5.3  Test  Cells 

2 . 5 . 3 . 1  Controlled  V  ariables 

2 . 5 . 3 . 2  Uncontrolled  Variables 

2.5.3.3  Sample  Sizes/Confidence  Levels/Technical  Significance 

2.5.4  Test  Activities 

2.5.4. 1  Test  Activity  1 

2.5.4.1.1  Relationship  of  test  activity  to  program 

2.5.4.1.2  Location 

2.5. 4. 1.3  Instrumentation  Requirements 

2.5.4. 1.4  Resource  Requirements 

2.5. 4. 1.5  Environmental  Considerations 

2. 5. 4. 2  Test  Activity  2 

2.5. 4.2.1  Relationship  of  test  activity  to  program 
2.5.4.22  Location 

2.5. 4.2.3  Instrumentation  Requirements 

2.5. 4.2.4  Resource  Requirements 

2.5. 4.2.5  Environmental  Considerations 
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2. 5. 4. 3  Test  Activity  n 

2.6  Schedule 

2.6.1  Test  Opportunities/Events 

2.6.2  JT &E  Schedule 

2.7  Resources  (size  may  dictate  that  content  be  included  as  an  annex) 

2.7.1  Consolidate  Resource  Estimate 

2.7.2  Fiscal  Requirements 

2.7.3  Alternatives 

2.7.3 . 1  Programmatic  Restructuring 

2.7. 3. 2  Technical  Restructuring 

3 . 0  Program  Analysis 

3 . 1  Analysis  Methodology 

3.1.1  Measure  Type  (percentages) 

3.1.2  Measure  Type  (expert  opinion) 

3 . 1 . 2 . 1  Expert  Selection 

3 . 1 . 2 . 2  Structure  Interview 

3.1. 2.2.1  Expert  Preparation 

3.1.2.2.2  Questionnaire 

3 . 1 . 2 . 3  Results  Analysis 

3.1.3  Measure  Type  (user  assessments) 

3.2  Description  of  Measures  (measure  attributes) 

3.2.1  Completeness 

3.2.2  Quality 

3.2.2. 1  Uncertainty 

3. 2.2. 2  Consistency 

3.2.3  Efficiency 

3.3  Event  Reconstruction 

4.0  JT&E  Products 

4.1  Description 

4.1.1  Documents 

4.1.2  Simulations,  Models 

4.1.3  Others 

4.2  Users  of  products 

4.3  Disposition  of  products 

5.0  JT&E  Management 

5.1  Organization 

5.2  Reports 

Annexes  (as  required  to  provide  rational  used  in  the  selection  of  methodologies,  techniques,  test 
methods,  test  activities,  etc.) 
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ANNEX  O 

TECHNICAL  ADVICE 


A.  INTRODUCTION 

The  JT&E  Program  receives  advice  from 
a  number  of  technical  organizations  that  are 
available  to  council  JT&E  Program  directors  and 
administrators,  Joint  Feasibility  Study  Directors, 
and  Joint  Test  Directors  on  technical,  test,  and 
doctrinal  issues  related  to  the  program  in  general 
and  specific  JT&Es  in  particular.  These  technical 
organizations  include  the  JT&E  Senior  Advisory 
Council  (SAC),  the  JT&E  Technical  Advisory 
Board  (TAB),  Technical  Advisory  Groups 
(TAG),  General  Officer  Steering  Committees 
(GOSC),  and  Functional  Expert  Panels.  The 
following  sections  contain  information  and 
recommendations  relative  to  when  an  advisory 
organization  could  be  beneficial,  who  is 
responsible  for  establishing/providing  advisory 
organizations,  and  how  these  advisory 
organizations  can  be  used  to  reduce  the  risk 
associated  with  JT&E  activities  and  events. 

B  .  JT&E  SENIOR  ADVISORY 

COUNCIL  (SAC) 

The  JT&E  SAC  is  a  senior-level  advisory 
group  that  reviews  candidates  for  entry  into  the 
JT&E  Program,  Joint  Feasibility  Study  and  Joint 
Test  and  Evaluation  progress  and  results,  and 
recommends  appropriate  actions  to  the 
D,T,SE&E.  The  Service  members  on  the  SAC 
also  commit  the  personnel  and  resources  required 
for  the  conduct  of  the  JFS  and  JT&E  activities. 
The  SAC  meets  in  June  of  each  year  or  at  the  call 
of  the  Chairperson.  The  following  organizations 
provide  personnel  who  are  permanent  members 
of  the  SAC.  A  list  of  current  SAC  members  is 
included  in  the  JT&E  Program  handout  entitled 
"Documents  Related  to  JT&E  Programs." 


The  JT&E  SAC  is  composed  of  the  following 
members: 


Co  chairperson 


Executive  Advisor: 
Technical  Advisor: 


D,T,SE&E  and  DOT&E 
OSD(PA&E) 

OSD(S&SS) 

Joint  Staff  (J-8) 

ODCSOPS-FDZ 

CNO(N-O91) 

OSD/C3I 
OSD/SOLIC 
USAF/TE 
USMC  Systems 
Command  -  Chief  of  Staff 
BMDO,  TMD 
Chairperson  of  JT&E 
Planning  Committee 
Chairperson  of  TAB  (non 
voting  member) 


The  responsibilities  of  the  JT&E  SAC  are 
as  follows: 


1.  Recommend  and  prioritize  candidates  for 

entry  into  the  JT&E  Program. 

2.  Review  JFS  results  in  terms  of: 

a.  Need  of  proposed  JT&Es. 

b.  Proposed  JT&E  concepts  and  Program 
Test  Designs. 

c.  Feasibility  of  proposed  JT&Es. 

d.  Impact  and  availability  of  resource 
requirements. 

e.  Availability  of  OSD  and  Service  funds  to 
support  the  proposed  JT&Es. 

f.  Recommendations  for  lead  and 
participating  Services. 

g.  Anticipated  improvements  in  joint 
capabilities. 
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h.  Opportunities  for  proposed  JT&Es  to  be 
incorporated  into  scheduled  joint 
activities. 

3.  Establish  schedules  for  JFSs  and  JT&Es. 

4.  Commit  Service  support  and  resources  to 
JFSs  and  JT&Es. 

5.  Review  and  recommend  JT&E  Program 
management  and  policy  changes. 

6.  Review  JT&E  final  reports. 

C .  JT&E  TECHNICAL  ADVISORY 
BOARD  (TAB) 

The  JT&E  TAB  is  an  organization  of 
senior  civilian  scientists  and  engineers  from  OSD 
and  the  Services  who  advise  the  SAC, 
nomination  sponsors,  FSDs,  and  JTDs  on  the 
technical  issues  related  to  the  JT&E  Program  in 
general  and  JFSs  and  JT&Es  in  particular.  The 
nomination  and  selection  of  TAB  members  is 
based  on  current  expertise  and  experience  relative 
to  the  planning,  conduct,  and  evaluation  of  large 
scale  tests,  exercises,  and  simulations.  The  TAB 
provides  advice  on  JT&E  nominations,  monitors 
JT&E  activities,  and  provides  technical  advice 
and  guidance  to  FSDs  and  JTDs  in  order  to 
minimize  the  risk  and  cost  associated  with 
JT&Es.  The  OSD  and  Services  nominate 
qualified  individuals  for  the  TAB.  The 

D, T,SE&E  reviews  the  nominations  for  JT&E 
TAB  membership  and  approves  those  considered 
to  be  most  qualified.  The  TAB  meets  in  May  of 
each  year  or  as  directed  by  the  Chairperson.  A 
list  of  current  JT&E  TAB  members  is  included  in 
the  JT&E  Program  handout  entitled  "Documents 
Related  to  JT&E  Programs." 

The  JT&E  TAB  is  composed  of  the 
following  persons: 

Chairperson:  Rotated  periodically  among 
permanent  members 


Permanent  members:  One  member  from  each  of 
the  Services 

Non  permanent  members:  Civilian  scientists 
having  specific  technical  expertise 
related  to  a  particular  JT&E  and 
supported/participating  CINC 
representatives,  as  appropriate 

The  responsibilities  of  the  JT&E  TAB  are 

as  follows: 

1.  Review  and  determine  the  technical  adequacy 
of  APAs  and  PTPs. 

2.  Review  and  determine  the  technical  adequacy 
of  individual  test  plans,  data  management  and 
evaluation  plans,  and  test  reports. 

3.  Provide  technical  recommendations  to  the 
JT&E  SAC  concerning  JT&E  Program 
candidates,  JFSs,  and  JT&Es. 

4.  Assist  in  defining  common  elements  among 
JT&Es,  maximizing  the  possibility  of 
resource  sharing,  minimizing  the  cost 
associated  with  JT&Es,  and  preserving 
significant  residual  assets  developed  in 
conjunction  with  JT&Es  for  future  use. 
These  residual  assets  are  primarily  scenarios, 
databases,  computer  simulations,  algorithms, 
test  facilities,  instrumentation,  and  threat 
simulators. 

5.  When  required,  recommend  personnel  to 
staff  TAGs  for  a  JFS  or  JT&E. 

6.  Provide  technical  advice  to  the  JT&E  SAC, 
JFSs,  and  JTDs. 

7.  Provide  recommendations  to  the  DDT&E 
relative  to  the  continuation  or  termination  of 
JFSs  and  JT&Es. 
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D .  TECHNICAL  ADVISORY  GROUP 

(TAG) 

A  TAG  is  a  technical  body  formed  to 
provide  direct  technical  support  to  a  JFS.  The 
FSD,  with  coordination  and  approval  of 
DDT&E,  will  determine  if  and  when  a  TAG  is 
required.  TAG  members  may  be  nominated  by 
the  TAB,  suggested  by  the  Service  Operational 
Test  Agencies  (OTAs),  requested  by  the  FSD,  or 
suggested  from  other  sources.  The  FSD  will 
coordinated  directly  with  the  Service  OTAs  to 
obtain  the  services  of  TAG  members.  The 
composition  of  a  TAG  may  vary  as  the  technical 
nature  of  the  JFS  changes.  The  TAG  will  meet  at 
the  call  of  the  FSD  and  is  not  a  permanent  full¬ 
time  part  of  the  JFS  staff.  The  TAG  may 
provide  assistance  either  collectively  or  through 
the  actions  of  individual  members. 

A  TAG  is  composed  of  the  following 
members: 

If  established,  the  TAG  composition  will  be 
determined  by  the  FSD  and  will  normally  be 
made  up  of  individuals  from  the  Services 
(usually  senior  civilians)  who  have  technical 
proficiency  particularly  suited  to  the  JFS  . 

The  responsibilities  of  a  TAG  are  as 
follows: 

1.  Review  and  provide  technical  comments  and 
recommendations  on  plans,  analysis  results, 
and  test  reports. 

2.  Assess  appropriateness  and  output  of 
simulations  suggested  or  developed  for  a 
proposed  JT&E  or  specific  test  events. 

3.  Identify  sources  of  technical  assistance  and 
provide  technical  liaison  with  these  sources. 


4.  Provide  the  JT&E  TAB  with  technical  advice 
and  assistance  as  required. 

5.  Provide  the  FSD  with  technical  advice  and 
assistance  as  required. 

E.  GENERAL  OFFICER  STEERING 

COMMITTEE  (GOSC) 

A  GOSC  should  be  established  to  review 
and  provide  advice  on  JFS  issues  pertaining  to 
policy,  doctrine,  tactics,  and  resources.  A 
GOSC  is  a  group  of  general  officers  from  the 
Services  and  participating  theater  CINCs  who  are 
invited  by  the  FSD  to  advise  on  doctrine,  policy, 
or  tactics  issues.  The  intent  of  a  GOSC  is  to 
capture  and  integrate  Service  representation  at  a 
senior  officer  level.  The  FSD  will  determine  if 
and  when  a  GOSC  is  required.  GOSC  members 
may  be  nominated  by  the  TAB,  suggested  by  the 
Service  OTAs,  requested  by  the  FSD,  or 
suggested  from  other  sources.  The  FSD  will 
coordinate  directly  with  the  Service  OTAs  to 
obtain  the  services  of  GOSC  members.  The 
GOSC  will  meet  at  the  call  of  the  FSD  and  is  not 
a  permanent  full-time  part  of  the  JFS  staff.  The 
GOSC  provides  guidance  either  collectively  or 
through  the  actions  of  individual  members  and 
should  be  included  in  the  review  cycle  of  JFS 
produced  concepts,  products,  and  documents. 

A  GOSC  is  composed  of  the  following 
members: 

If  established,  the  GOSC  composition  will  be 
determined  by  the  FSD  and  will  normally  be 
made  up  of  flag  officers  (1-2  star  level)  from 
the  Services  involved  in  the  JFS,  those  who 
will  provide  test  resources,  or  those  who 
could  be  affected  by  the  JFS  results  and 
recommendations. 
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The  responsibilities  of  a  GOSC  are  as 
follows: 

1.  Provide  the  FSD  with  a  forum  for  senior  level 
advice  and  guidance  with  respect  to  Service 
policy,  doctrine,  and  tactics. 

2.  Review  APA,  study  results/conclusions,  and 
reports. 

3.  Provide  advice  relative  to  the  availability  or 
resources  and  Service  events  to  support 
planned  activities. 

4.  Provide  advice  relative  to  coordination 
requirements  and  the  identification  of 
signature  authorities. 

5.  Provide  assistance  in  arranging  meeting  and 
briefing  schedules. 

6.  Provide  assistance  in  obtaining  the  use  of 
Service  assets  and  resources  to  include 
information  or  access  to  classified  programs. 

F.  FUNCTIONAL  EXPERT  PANELS 

If  the  JFS  involves  issues,  techniques,  or 
equipment  that  are  technically  complex,  the  FSD 
should  consider  establishing  Functional  Expert 
Panels  to  obtain  expertise  or  assistance  that  is  not 
available  within  the  JFS.  Functional  Expert 
Panels  will  involve  the  technical  communities 
associated  with  the  JFS  activities,  and  this 
involvement  will  be  helpful  in  the  coordination 
and  acceptance  of  JFS  results.  Functional  Expert 
Panel  members  may  be  suggested  by  the  TAB, 
the  Service  OTAs,  other  sources,  or  identified  by 
the  FSD.  The  FSD  will  coordinate  with  the 
Service  OTAs  to  obtain  the  services  of  Functional 
Expert  Panel  members  from  Service  resources. 
The  FSD  will  coordinate  with  the  OSD  JT&E 
Coordinator  to  obtain  the  services  of  non-DoD 
Functional  Expert  Panel  members. 

A  Functional  Expert  Panel  is  composed 
of  the  following  members: 


If  established,  the  Functional  Expert  Panel 
composition  will  be  determined  by  the  FSD 
and  will  be  made  up  of  members  who  are 
experts  in  the  required  specific  test  areas. 
Functional  Expert  panel  members  can  be 
Service  members,  Government  Service 
employees,  contractors,  or  in  some  instances, 
civilians. 

The  responsibilities  of  a  Functional 
Expert  Panel  are  as  follows: 

1.  Provide  direct  technical  expertise  and 
assistance  to  the  JFS. 

2.  Leverage  JFS  expertise  by  involving  the 
technical  communities. 

3.  Assist  in  identifying  and  obtaining  additional 
expertise  when  required. 

4.  Expand  awareness  of  JFS  efforts  to  the 
technical  communities. 

5.  Assist  in  coordination  and  acceptance  of  JFS 
results  and  reports. 

6.  Assist  in  avoidance  of  the  "not  invented  here" 
syndrome. 

G .  JT&E  PLANNING  COMMITTEE 

(JT&E  PC) 

The  JT&E  PC  is  a  working  group  that 
assists  in  developing  and  evaluating  JT&E 
nominations  and  preparing  them  for  senior  level 
review.  The  JT&E  PC  Chairperson  is  the  OSD 
JT&E  Coordinator.  The  JT&E  PCs  permanent 
membership  is  drawn  from  OSD,  the  Joint  Staff, 
and  the  Services.  The  JT&E  PC  members 
interface  with  and  represent  their  respective 
Services  and  agencies.  The  Services  and 
agencies  may  send  additional  personnel  to  the 
JT&E  PC  meetings  to  provide  additional 
information  or  assistance  as  may  be  required. 
The  JT&E  PC  meets  at  the  call  of  the 
Chairperson  and  serves  as  the  action  staff  for  the 
JT&E  TAB  and  SAC. 
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The  JT&E  PC  is  composed  of  the 
following  members: 

Chairperson:  OSD  JT&E  Coordinator 
Permanent  members:  One  member  from  DOT&E 
One  member  from  the  Joint  Staff 
One  member  from  each  of  the 
Services 

One  member  from  OSD/P A&E 

Non  permanent  members:  One  representative 
from  each  agency  submitting  a 
nomination  for  consideration 


The  responsibilities  of  the  JT&E  PC  are 

as  follows: 

1.  Receive  and  prioritize  JT&E  Program 
nominations. 

2.  Review  JT&E  nominations  to: 

a.  Determine  jointness,  feasibility,  and 
validity  of  potential  results. 

b.  Identify  duplications  and  possible 
consolidations. 

c.  Determine  adequacy  of  data,  resource 
estimates,  and  test  objectives. 

3.  Recommend  nominations  by  priority  to  the 
TAB  for  consideration. 

4.  Functions  as  the  action  staff  for  the  JT&E 
TAB  and  SAC. 
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